# Summary of piping/chaining proposal?

**URL:** <https://discourse.julialang.org/t/summary-of-piping-chaining-proposal/91442>\
**Category:** Internals & Design\
**Created:** [December 8, 2022, 7:48pm UTC](https://discourse.julialang.org/t/summary-of-piping-chaining-proposal/91442 "2022-12-08T19:48:39Z")\
**Posts on this page:** 1\
**Showing post:** 8

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [December 19, 2022, 1:30pm UTC](https://discourse.julialang.org/t/summary-of-piping-chaining-proposal/91442/8 "2022-12-19T13:30:44Z")

</div>

> [@qsong](#):
>
> IMHO, the proposed curly braces syntax `{f, g}` allow people to legally write
> 
> ```julia-auto
> x |> {f(a, b, _,c), g(_, d, kwd=e)},
> 
> ```

Almost. My [third proposal](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue-rev-3/90836) is to make this legal:

```julia
x |> {f(a, b, it, c); g(it, d; kwd=e)}

```

or, in order to avoid compiling an unnecessary lambda and potentially suffer performance loss from its [variable capture behaviors](https://github.com/JuliaLang/julia/issues/15276) (and to have better [tighter] operator precedence),

```julia
x.{f(a, b, it, c); g(it, d; kwd=e)}

```

The key thing to note is that I’m _explicitly avoiding_ claiming the `_` character for the chaining syntax, because I think it serves very well for denoting partial application (which is a distinct concept from the `it` keyword, and would be very useful outside the bounds of `{...}`).

If this proposal is accepted _and_ [PR#24990](https://github.com/JuliaLang/julia/pull/24990) is accepted, then what you have written will be valid.

* * *

* * *

> [@devel-chm](#):
>
> I suggest we should keep using independent packages with macros to  
> implement candidate chaining/piping/… syntax options until a community  
> consensus on the best approach is reached.  
> …  
> Lets try them all out against eachother for a year to get things  
> stable and robust and then revisit what makes the most sense  
> for Julia going forwards.

Considering the specifics of this problem (which I laid out in my [first proposal](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654)), this doesn’t seem to be the best decision-making process here. I would propose instead _accepting a chaining syntax[es] on a probationary basis for some period_, i.e., with no assurance that the syntax[es] will continue to be part of the language after that period, and choosing in the end whether to keep it.

I do think we want to spend some more time and experience to make sure it’s robust and stable; I just don’t think that developing consensus without substantial firsthand experience _as a language feature_ is meaningful; the signal-to-noise ratio there would be pretty low.

I will also note that the dominant chaining packages are actually not particularly conducive to the genericism desired of a language feature anyway; for example, semantics like in Chain.jl, which automatically threads into first argument position when `_` isn’t specified, is convenient for `DataFrame`s but isn’t particularly desirable for functional styles (for example, when using currying functions like `filter(f::Function)` or when using [transducers](https://clojure.org/reference/transducers) as part of a chain); meanwhile DataPipes.jl, which automatically threads into last argument position, isn’t helpful for functions written in an object-oriented style like those for `DataFrame`s, nor for the curried binary operators like `>(5)`; these behaviors therefore take away from the genericism and composability you want of a proper language feature.

> **Example of Chain.jl poor behavior with Transducers**
>
> ```julia
> julia> using Chain, Transducers
> 
> julia> @chain collect(1:100) begin
> Map(x->2x)
> Filter(>(100))
> sum
> end
> ERROR: MethodError: no method matching Map(::Vector{Int64}, ::var"#3#4")
> 
> julia> @macroexpand@chain collect(1:100) begin
> Map(x->2x)
> Filter(>(100))
> sum
> end
> quote
> local var"##356" = collect(1:100)
> #= REPL[65]:2 =#
> local var"##357" = Map(var"##356", (x->begin
> #= REPL[65]:2 =#
> 2x
> end))
> #= REPL[65]:3 =#
> local var"##358" = Filter(var"##357", (>)(100))
> #= REPL[65]:4 =#
> local var"##359" = sum(var"##358")
> var"##359"
> end
> 
> ```

> **Example of DataPipes.jl poor behavior with curried operator**
>
> ```julia
> julia> using DataPipes
> 
> julia> @p begin
> 10
> >(5)
> end
> ┌ Warning: Pipeline step top-level function is an operator. An argument with the previous step results is still appended.
> │ func = ">"
> │ args =
> │ 1-element Vector{Int64}:
> │ 5
> └ @ DataPipes C:\Users\unime\.julia\packages\DataPipes\z06K1\src\pipe.jl:257
> false
> 
> julia> @macroexpand@p begin
> 10
> >(5)
> end
> ┌ Warning: Pipeline step top-level function is an operator. An argument with the previous step results is still appended.
> │ func = ">"
> │ args =
> │ 1-element Vector{Int64}:
> │ 5
> └ @ DataPipes C:\Users\unime\.julia\packages\DataPipes\z06K1\src\pipe.jl:257
> quote
> #= C:\Users\unime\.julia\packages\DataPipes\z06K1\src\pipe.jl:47 =#
> #= REPL[58]:2 =#
> var"##res#345" = 10
> #= REPL[58]:3 =#
> var"##res#346" = 5 > var"##res#345"
> #= C:\Users\unime\.julia\packages\DataPipes\z06K1\src\pipe.jl:48 =#
> var"##res#346"
> end
> 
> ```

> **How these should work**
>
> ```julia
> julia> using MethodChains, Transducers
> 
> julia> MethodChains.init_repl()
> 
> julia> (10).{>(5)}
> true
> 
> julia> collect(1:100).{Map(x->2x);Filter(>(100)),sum}
> 7550
> 
> julia> @macroexpand collect(1:100).{Map(x->2x);Filter(>(100)),sum}
> :(let it = collect(1:100)
> it
> it = (Map((x->begin
> #= REPL[158]:1 =#
> 2x
> end)))(it)
> it = (Filter((>)(100)))(it)
> it = sum(it)
> it
> end)
> 
> ```

> [@devel-chm](#):
>
> I particularly dislike the use of “{” or “}” and the introduction of complex  
> syntax into julia and locking in these extra bracketing characters into  
> the language proper.

So far, I haven’t found any good reasons to resonate with this sentiment. Is it merely a protest over aesthetics? Also, there are many things in Julia \*much\* more complex than what I have proposed, which seems inevitable considering Julia’s target audiences.

We have only three sets of bracing characters available on our keyboards: parentheses `()`, square brackets `[]`, and curly braces `{}` (four if you count angle brackets `<>`, but we put those to very good uses already). Banishing one of them from a desire for Python-zen seems non-Julian.

Julia chose to use Algol-like `begin...end` for block expressions, which was a wonderful decision because it made it more natural to use the same style for other blocks (e.g. `let`, `if`, etc.) and freed up `{}` for other things.

When I consider the uses for `{}`, they seem to be primarily for denoting unordered lists (for which we have `Set()` and don’t find useful enough to justify dedicated syntax), or for set-builder notation, or for denoting switching expressions (for which we have `if...elseif`), or for blocks of expressions (for which we have `begin...end` and `(...; ...)`). Compared with the hypothetical uses pondered [here](https://discourse.julialang.org/t/why-begin-end-instead-of-for-block-delimiter/89735/15), using it for function chains seems the most interesting and useful.

Whereas `(x,y,z)` is great for assembling a collection of objects and `f(x,y,z)` for calling a function on it, `{f,g,h}` is nice for assembling a collection of functions and `x.{f;g;h}` for passing an object through it. It’s hard to ignore the beauty in the symmetry, if even with the `.` dot; it’s akin to the symmetry in Julia’s decision that functions be objects and objects be functions.

* * *

* * *

@mikmoore I wasn’t smart enough to understand FixArgs.jl, so in my [second proposal](https://discourse.julialang.org/t/fixing-the-piping-chaining-partial-application-issue-rev-2/90408) I developed my own partial applicator type (which is [accessible here](https://github.com/uniment/ChainingDemo.jl)), which I [showed](https://discourse.julialang.org/t/fixing-the-piping-chaining-partial-application-issue-rev-2/90408/2) to have favorable performance compared to `Base.Fix1` and `Base.Fix2`.

I made mine to allow nonfixed kwargs to override fixed, and allow from-end-positional arguments to be fixed; it works like `Fix{positions, num_of_args}(fun, fixedargs...; fixedkwargs...)`, where `num_of_args` dictates the number of arguments in the final call (and a `-1` value indicates varargs). You can run it like this:

```julia
julia> using ChainingDemo

julia> Fix{(1, 3), 3}(f, 1, 3; a=1, c=3)
f(1, _, 3, ; a=1, c=3)

julia> @underscores f(1, _, 3; a=1, c=3)
f(1, _, 3, ; a=1, c=3)

julia> FixFirst(f, "hi!")
f(hi!, _...)

julia> FixFirst(f, "hi!") |> typeof
FixFirst{typeof(f), String, NamedTuple{(), Tuple{}}} (alias for Fix{typeof(f), (1,), -1, Tuple{String}, NamedTuple{(), Tuple{}}})

julia> Fix{(1,-3,-1), -1}(f, :start, :nextnextlast, :last)
f(start, _..., nextnextlast, _, last)

julia> @underscores f(:start, _..., :nextnextlast, _, :last)
f(start, _..., nextnextlast, _, last)

julia> @underscores filter(_%3==0, 0:10)
4-element Vector{Int64}:
 0
 3
 6
 9

julia> @underscores (xs = (x=>x^2 for x ∈ 1:4); map(_[2]/2, xs))
4-element Vector{Float64}:
 0.5
 2.0
 4.5
 8.0

```

> Note: from-end indices are allowed _only_ with varargs.

> Also note: if you try it now, pretty-printing doesn’t work in the REPL because I made `Fix` subtype `Function`; in the REPL, objects that subtype `Function` have their `Base.show` overridden, so you need to call `show` manually.

I agree that a generalized `Fix` type would be very useful, and I think PR#24990 would be very nice syntax sugar for this. I think these would be incredibly helpful in many contexts, most notably when used with `filter` and `map`.

However, I don’t think that this combined with `|>` should be the preferred way to make chains, because using a partial applicator in a chain constructs a partial functor that will be used just once and discarded. This is wasteful for compile time and memory. And as I’ve also opined, `|>` is awkward and has the wrong precedence for most uses.

Notice that I made `_` work as part of a partial application syntax like PR#24990; I do still like it and want it.

* * *

* * *

@MattEri

> **referenced code**
>
> > [@MattEri](#):
> >
> > I kinda dislike the type of code this leads (me) to, when writing actual pipelines. Let me illustrate with a simple example. (Feel free to provide corrections or better style!)
> > 
> > ```julia-auto
> > process_list = list ->
> > list.{
> > map(convert(Float32, {it}), it),
> > filter({it > 0}, it)
> > }
> > 
> > ```

Your example is incorrect, and should instead be:

```julia
process_list = {
    map({convert(Float32, it)}, it)
    filter({it > 0}, it)
}

```

and if PR#24990 were accepted, and using the curried form of `filter` available in Julia 1.9 ~~(and supposing a partially-applied form of `map` were to become available too, e.g. `map(f::Function) = FixFirst(map, f)`)~~, it could soon read like this:

```julia
process_list = { map(convert(Float32, _), _), filter(_>0) }

```

> [@MattEri](#):
>
> Front/back passing

This is the same concept as my first proposal, except using claimed syntax which would require parser changes and break many things. My first proposal chose `/>` and `\>` (which are currently invalid and thus unclaimed syntax) to avoid these problems.

> [@MattEri](#):
>
> Shorthand lambdas:

We cannot claim `it` outside of `{}` because it’s a valid identifier frequently used for iterators, and therefore claiming it would be a hugely breaking change. The nice thing about underscore `_` is that using it as an `rvalue` is unclaimed syntax (yet it parses as an identifier), which makes it super interesting: it can be claimed outside special braces without causing a breaking change.

However, we must draw a distinction between the concept of a “quick lambda” and the concept of “partial application.” The debate of PR#24990 has persisted for years because of a desire to use `_` to build “quick lambdas” which can do more than just partially-apply a single function. The problem with this idea is that, at the parser level, it’s generally impossible to tell where the bounds of such a “quick lambda” should be (the parser is unfortunately not a mind reader, even if Julia makes it seem so).

This makes the desire to form “quick lambdas” purely on the basis of using a special identifier untenable. ~~That said, for most of the things that a “quick lambda” is wanted for, i.e. a single argument that passes through a couple simple functions (and never reaching a reducing function which combines it with itself in any way, nor encountering any branching logic), the partial applicator functor can work if combined with a function composition fallback as [described here](https://github.com/JuliaLang/julia/pull/24990#issuecomment-350728180).~~

~~For example, this works with the demo code of my second proposal, mentioned above:~~

> ****
>
> ```julia
> julia> using ChainingDemo
> 
> julia> @underscores @show g = √(2_+3) > 5;
> g = √(Fix{(1,), 2}(*, 2) + 3) > 5 = >(_, 5) ∘ sqrt(_) ∘ +(_, 3) ∘ *(2, _)
> 
> julia> g(10)
> false
> 
> julia> @underscores filter(√(2_+3) > 5, 5:15)
> 4-element Vector{Int64}:
> 12
> 13
> 14
> 15
> 
> ```

However, for more general “quick lambdas” where the object takes more than one path and it interacts with itself, or if there will be branching logic, you need some unclaimed syntax to set the bounds of the expression within which the identifier has the desired meaning. Conveniently, my third proposal defines such bounds with `{}`, allowing you to construct arbitrary quick lambdas using `it` as an identifier. For example, `{it+it^2}` constructs a function that works like `it->it+it^2`.

Also notable, when you consider the use of “it” in English, it is for exactly this purpose of defining how an object should relate to itself (as well as for argument threading).

* * *

* * *

## Summary

I think that in a perfect world we would have:

1. A `Fix` functor type for generalized partial application (e.g. my [demo code](https://github.com/uniment/ChainingDemo.jl) or FixArgs.jl)
2. Use of `_` underscores in a partial application syntax, à la PR#24990 (or my demo code)
3. ~~A function composition fallback for partial functions when passed as arguments to other functions. [As mentioned in the PR](https://github.com/JuliaLang/julia/pull/24990#issuecomment-350728180), that might be tricky; I don’t currently have a way to judge how tricky.~~
4. Implementation of my [third proposal](https://github.com/uniment/MethodChains.jl), to use `{}` for a chaining syntax. Out of my three proposals, I maintain that this one is the best.

At least in this case, I suspect the path to making the optimal decision is different from the optimal path to making a decision. Accepting these ideas into the language on a probationary basis seems appropriate (after some more vetting, certainly).

---

_[View the full topic](https://discourse.julialang.org/t/summary-of-piping-chaining-proposal/91442)._