# \[RFC\] PipelessPipes.jl (now Chain.jl)

**URL:** <https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448>\
**Category:** Package Announcements\
**Created:** [November 19, 2020, 4:25pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448 "2020-11-19T16:25:38Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [November 19, 2020, 4:25pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/1 "2020-11-19T16:25:38Z")

</div>

[PipelessPipes.jl](https://github.com/jkrumbiegel/PipelessPipes.jl) is not a package, yet, as I would be interested in hearing some opinions first. Here are my thoughts about Pipe.jl and Lazy.jl, which I have tried previously:

## Pipe.jl

### pros

- \_ syntax is nice and non-magic

### cons

- I don’t like typing `|>`
- The last entry of a pipe can’t have a trailing `|>`, making it harder to comment out parts
- If there’s an error in one part, the whole expression errors and VSCode can’t highlight the specific part
- end of pipeline just “dangles” in terms of indentation, which means the IDE always tries to indent also the following lines
- you can’t really interject some random statement for checking what’s going on inside the pipe that well

## Lazy.jl’s `@>`:

### pros

- begin / end is nice for indentation after
- no `|>` necessary

### cons

- Not every function needs piped thing as first argument, like `filter` (also not necessarily last, the other option `@>>`)
- Looks a bit too magic for me when the first argument is not visible at all

## PipelessPipes.jl

**edit: As I’ve made some changes since the first post, and have registered those changes now, here’s the relevant excerpt from the readme**

PipelessPipes defines the `@_` macro. It takes a start value and a `begin ... end` block of expressions.

The result of each expression is fed into the next one using one of two rules:

1. **There is at least one underscore in the expression**

- every `_` is replaced with the result of the previous expression

1. **There is no underscore**

- the result of the previous expression is used as the first argument in the current expression, as long as it is a function call or a symbol representing a function.

Lines that are prefaced with `@!` are executed, but their result is not fed into the next pipeline step.  
This is very useful to inspect pipeline state during debugging, for example.

### Example with a DataFrame:

```julia
using DataFrames, PipelessPipes

df = DataFrame(group = [1, 2, 1, 2], weight = [1, 3, 5, 7])

result = @_ df begin
    filter(r -> r.weight < 6, _)
    groupby(:group)
    @! println("There are $(length(_)) groups after step 2.")
    combine(:weight => sum => :total_weight)
end

```

The pipeless block is equivalent to this:

```julia
result = let
    var1 = filter(r -> r.weight < 6, df)
    var2 = groupby(var1, :group)
    println("There are $(length(var2)) groups after step 2.")
    var3 = combine(var2, :weight => sum => :total_weight)
end

```

For debugging, it’s often useful to look at values in the middle of a pipeline.  
You can use the `@!` macro to mark expressions that should not pass on their result.  
For these expressions there is no implicit first argument spliced in if there is no `_`, because that would be impractical for most purposes.

For the error message point, look at the difference to `Pipe.jl` here from some random code of mine:

PipelessPipes.jl

 ![grafik](https://global.discourse-cdn.com/julialang/original/3X/1/1/11de03122127b5b6c1d7bdb5022b589bfefc089c.png)

Pipe.jl:

 ![grafik](https://global.discourse-cdn.com/julialang/original/3X/3/f/3fccab8955ebbe8e61e0a05d6e8458a794a32c67.png)

### Design Questions

I wonder what the best name for the macro is, because all the good ones are already taken by other packages. I like `@_` because it’s really short, although it would conflict with Underscores.jl, if one wanted to use both.

The other question is if this ruleset has some obvious problems to you. I haven’t found serious issues so far, but maybe I’ve overlooked some circumstances under which this expression parsing would break.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [November 19, 2020, 4:29pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/2 "2020-11-19T16:29:34Z")

</div>

It’s good to see more experimentation in the Piping space. Here are two thoughts

1. I t would be great if you could optionally omit the `_` and have that default to the first argument. But then override that if the user provides `_` anywhere
2. Having a trailing `|>` can be annoying, but if you are copying and pasting to the REPL it’s easier to use `|>` because otherwise you need to add an `end` which is annoying.

It would be great if it came in a “block” form and a “magrittr” form.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [November 19, 2020, 4:35pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/3 "2020-11-19T16:35:02Z")

</div>

My first thought was very against this. I’ve reread it and now I kind of like it. I do use Underscores.jl more and more often, and for things that aren’t necessarily data pipelines, so I’m not a fan of the conflicting symbols unless you are going to provide a somewhat substantial replication of its behavior. How does your proposal fair outsides of DataFrames usage?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [November 19, 2020, 5:04pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/4 "2020-11-19T17:04:20Z")

</div>

Added to the big list  
[https://github.com/JuliaLang/julia/issues/5571#issuecomment-205754539](https://github.com/JuliaLang/julia/issues/5571#issuecomment-205754539)

---

<div class="post-metadata">

**Author:** ![Marc.Cox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marc.cox/32/7514_2.png) [@Marc.Cox](https://discourse.julialang.org/u/Marc.Cox)\
**Post date:** [November 19, 2020, 8:21pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/5 "2020-11-19T20:21:29Z")

</div>

**Julius @jules ,** Thank You very much for this RFC around enhancing the previous Linux/Unix Pipe syntax which is a very helpful foundation and a natural fit for distributed processing, but in this iteration I hope we can elevate (or is that continue ? See below \*\***)** the language syntax to a slightly higher abstraction layer using **mathematical notation,** while simultaneously keeping the super fast and efficient **Automatic vectorization** gears hidden/encapsulated to achieve this \>\> [Automatic vectorization - Wikipedia](https://en.wikipedia.org/wiki/Automatic_vectorization)

So also request support syntax for Function Composition (computer science) described here \>\>

> **[Function composition (computer science)](https://en.wikipedia.org/wiki/Function_composition_%28computer_science%29)**
>
> In computer science, function composition is an act or mechanism to combine simple functions to build more complicated ones. Like the usual composition of functions in mathematics, the result of each function is passed as the argument of the next, and the result of the last one is the result of the whole.
> Programmers frequently apply functions to results of other functions, and almost all programming languages allow it. In some cases, the composition of functions is interesting as a function in ...

" _The ability to_ _**easily compose functions encourages [factoring](https://en.wikipedia.org/wiki/Code_refactoring) (breaking apart) [functions](https://en.wikipedia.org/wiki/Subroutine) for maintainability and [code reuse](https://en.wikipedia.org/wiki/Code_reuse). More generally, big systems might be built by composing whole programs**_ **."**

And so here is another example in a similar proposal that seems to have come out of some previous RFC / design work :

It’s especially important ( to provide an alternative to current **|\>** syntax ) when you are using Julia for data analysis, where you commonly have data transformation pipelines.

In particular, Pandas in Python is convenient to use because you can write things like _**df.groupby(“something”).aggregate(sum).std().reset\_index()**_ , which is **a**  **nightmare to write with the current |\> syntax.**

@@ [Function chaining · Issue #5571 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/5571#issuecomment-33437023)

\*\*(or continue ? See this \>\> **)** IF this \>\> [My mental load using Julia is much higher than, e.g., in Python. How to reduce it? - #5 by rdeits](https://discourse.julialang.org/t/my-mental-load-using-julia-is-much-higher-than-e-g-in-python-how-to-reduce-it/18902/5) \<\< is already implemented as it appears to be , it would already address what I’m requesting here per this statement :

I think what you’re talking about is “fluent interfaces”, and it’s true that we don’t really do that in Julia, at least not in the same way. You might find the `|>` operator useful, since you can do:

```julia

> x |> f |> g

```

as an alternative notation for g(f(x)) **.**

**.**

**However**  **as**  **@rdeits has previously**  **NOTED there is a**  **“missing” functionality**  **\>\>** Notably, in a fluent interface, there is no way to chain a function that _isn’t_ a method of the returned object. So if **sort()** is a generic function (and not a method of whatever drawInContext returns), then **you can’t do:**

```julia
	line.mirror.drawIncontext(ctx).sort()

```

So if all ^the above^ has been said and done and the `g(f(x))` `notation `is  
in place , then for this RFC I’d like the proposed **new** _ **PipelessPipes.jl** _ Package Design to be mostly about Julia compiler / LLVM optimizations similar to [JuliaFolds · GitHub](https://github.com/JuliaFolds) for optimizing compilation of the new **functional composition syntax** to **automatically**  **evaluate**  **how**  **to**  **/if** can SIMD / Vectorize \*\* / Parallelize / Thread the Nested functions – As a starter for more background consider searching reading about SIMD designs \>\>

> **[Single instruction, multiple data](https://en.wikipedia.org/wiki/SIMD)**
>
> Single instruction, multiple data (SIMD) is a type of parallel processing in Flynn's taxonomy. SIMD can be internal (part of the hardware design) and it can be directly accessible through an instruction set architecture (ISA), but it should not be confused with an ISA. SIMD describes computers with multiple processing elements that perform the same operation on multiple data points simultaneously.
> Such machines exploit data level parallelism, but not concurrency: there are simultaneous (parallel...

The hardware handles all alignment issues and “strip-mining” of loops. Machines with different vector sizes would be able to run the same code.[[6]](https://en.wikipedia.org/wiki/SIMD#cite_note-6) **Clang LLVM** ( used by Julia which ) calls this vector type “ **vscale** ”.

-and-

[https://discourse.julialang.org/search?q=SIMD](https://discourse.julialang.org/search?q=SIMD)

Re: Existing Vectorize syntax \*\* ( highly desirable to keep compatible with proposed new **functional composition syntax** Package Design )

[https://docs.julialang.org/en/v1/manual/functions/#man-vectorized](https://docs.julialang.org/en/v1/manual/functions/#man-vectorized)

`HTH`

`Ps` `>` `CCing Some others in for possible ` `interest or ` `additional insight ` `(like is this possible ? LOL ; and would it help them` `)` `:`  
@rdeits@MikeInnes @tkf

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [November 19, 2020, 8:28pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/6 "2020-11-19T20:28:03Z")

</div>

> [@jules](#):
>
> ## Lazy.jl’s @\>:
> 
> […]
> 
> ### cons
> 
> - Not every function needs piped thing as first argument, like `filter` (also not necessarily last, the other option `@>>` )
> - Looks a bit too magic for me when the first argument is not visible at all

It seems to me that these disadvantages only apply (as you noted) specifically to `@>` and `@>>`, but not really to the third threading macro introduced by [`Lazy.jl`](https://github.com/MikeInnes/Lazy.jl): the `@as` macro. For example:

```julia
julia> using Lazy: @as
julia> using DataFrames

julia> df = DataFrame(size=10*rand(10), weight=100*rand(10), color=rand([:red, :green], 10), shape=rand([:square, :circle], 10));

julia> @as _it_ df begin
           filter(r->r.size>1, _it_)
           groupby(_it_, [:color, :shape])
           combine(_it_, :weight => sum => :total_weight)
       end
4×3 DataFrame
 Row │ color shape total_weight 
     │ Symbol Symbol Float64      
─────┼──────────────────────────────
   1 │ red circle 191.114
   2 │ red square 8.4176
   3 │ green circle 154.489
   4 │ green square 146.474

```

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [November 20, 2020, 3:23am UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/7 "2020-11-20T03:23:00Z")

</div>

looks like a good candiate. release it as a package soon.

personally, for now I just use Lazy.jl’s @\>

---

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [November 20, 2020, 4:45am UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/8 "2020-11-20T04:45:54Z")

</div>

This is exactly what I’ve been waiting for. Thanks for working this out!

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [November 20, 2020, 12:19pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/9 "2020-11-20T12:19:11Z")

</div>

> [@pdeffebach](#):
>
> I t would be great if you could optionally omit the `_` and have that default to the first argument. But then override that if the user provides `_` anywhere

In my latest revision I’m doing that as I do agree that it saves quite some visual clutter for the most common cases.

I’ve also changed the rule with the double underscore, because I think it didn’t communicate well that there was a special line in the pipeline that shoudn’t affect the result.

Now there is a `@!` fake-macro that when prefacing a line disables the auto-first-arg insertion and also takes the line out of the pipeline. The \_ still works. So one can write the upper example like this, note the inserted `display` command. I find things like that very useful for debugging, your mileage may vary 😉

```julia
result = @_ df begin
    filter(r -> r.trial_type === "looped-audio-image-form", _)
    select([:subject, :checkboxes])
    groupby([:subject, :checkboxes])
    combine(:checkboxes => ByRow(extract_checkboxes) => AsTable)
    @! display(first(_, 5)) # have a peek at the first 5 rows here for debugging
    select(Not(:checkboxes))
    flatten([:group, :image, :checked])
    groupby([:group, :image])
    combine(:checked => sum)
    sort([:group, order(:checked_sum, rev = true)])
    groupby(:group)
end

```

The nice side effect is that one is forced to `@!` mark every line where one is not using the replacement mechanism, either implicit first argument or explicit with `_`. That means there is less danger of sticking a line in there that influences the result unexpectedly.

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [November 20, 2020, 12:59pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/10 "2020-11-20T12:59:27Z")

</div>

I was sure I had seen something like this already, and actually Lazy.jl had the `@_` macro. It looks like it was introduced [here](https://github.com/MikeInnes/Lazy.jl/commit/e4fedfc85db6604d667a0c2e2153723c1c116917). I’m not completely sure why it was removed (maybe it had issues with the `all-underscore identifier used as rvalue` deprecation), but I’m glad it came back in PipelessPipes.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [November 20, 2020, 3:13pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/11 "2020-11-20T15:13:05Z")

</div>

Registered it now, and updated the first post to reflect that

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [November 20, 2020, 3:47pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/12 "2020-11-20T15:47:38Z")

</div>

Nice! With the addition of implicit-first-argument piping, I think this implements all the features that I want in a piping macro. I [argued for something similar](https://github.com/JuliaData/DataFramesMeta.jl/issues/177#issue-709801971) in DataFramesMeta.

This is slightly off-topic, but when I do use Pipe.jl or Hose.jl, I like to write the pipes like this:

```julia
using DataFrames.jl
using Hose.jl

df = DataFrame(a = repeat(1:5, outer = 20),
               b = repeat(["a", "b", "c", "d"], inner = 25),
               x = repeat(1:20, inner = 5))

out = @hose (
    df
    |> @transform(y = 10 * :x)
    |> @where(:a .> 2)
    |> @by(:b, meanX = mean(:x), meanY = mean(:y))
    |> @orderby(:meanX)
    |> @select(:meanX, :meanY, var = :b)
)

```

There is less noise from the piping operators when they are all lined up. 🙂

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [November 20, 2020, 4:03pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/13 "2020-11-20T16:03:13Z")

</div>

Absurd how few lines of code are necessary to make a package like this! Was surprised when I looked at the source.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [November 20, 2020, 4:30pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/14 "2020-11-20T16:30:33Z")

</div>

> [@jules](#):
>
> I wonder what the best name for the macro is, because all the good ones are already taken by other packages.

I guess I’m a little late with this comment, since you’ve already registered the package, but `@chain` is available. If you named the package Chain.jl, then we would have Pipe.jl, Hose.jl, and Chain.jl. 🙂

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [November 20, 2020, 5:19pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/15 "2020-11-20T17:19:02Z")

</div>

> [@tbeason](#):
>
> Absurd how few lines of code are necessary to make a package like this! Was surprised when I looked at the source.

Yeah no kidding! A better programmer than me would probably need even less, but I tend towards legible verbosity 😉

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [November 20, 2020, 5:19pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/16 "2020-11-20T17:19:46Z")

</div>

> [@CameronBieganek](#):
>
> I guess I’m a little late with this comment, since you’ve already registered the package, but `@chain` is available. If you named the package Chain.jl, then we would have Pipe.jl, Hose.jl, and Chain.jl.

That’s also a good idea, technically I still have 3 days to change it 🤷‍♂️

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [November 21, 2020, 3:15pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/17 "2020-11-21T15:15:47Z")

</div>

**PipelessPipes** is not a bad name, but I think **Chain** is more pithy. Also, `@chain` is more descriptive than `@_`. I vote for changing the name to Chain.jl. 😃

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [November 21, 2020, 4:06pm UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/18 "2020-11-21T16:06:30Z")

</div>

Yeah I actually agree with that, I think `@chain` is not too bad to type either, I couldn’t think of something similarly short at the time. I’ll see if I can change it

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [November 23, 2020, 3:04am UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/19 "2020-11-23T03:04:01Z")

</div>

> [@CameronBieganek](#):
>
> then we would have Pipe.jl, Hose.jl, and Chain.jl.

Personally, I would just use Chain.jl once all the bugs are fixed.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [November 23, 2020, 4:10am UTC](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448/20 "2020-11-23T04:10:48Z")

</div>

The presumptive bugs or all the bugs you found already? 🐞🐛🐜

[Next page](https://discourse.julialang.org/t/rfc-pipelesspipes-jl-now-chain-jl/50448.md?page=2)