# Fixing the Piping/Chaining Issue (Rev 3)

**URL:** <https://discourse.julialang.org/t/fixing-the-piping-chaining-issue-rev-3/90836>\
**Category:** Internals & Design\
**Tags:** multithreading, syntax, piping, chaining, threading\
**Created:** [November 26, 2022, 2:50am UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue-rev-3/90836 "2022-11-26T02:50:46Z")\
**Posts on this page:** 1\
**Showing post:** 1

<div class="post-metadata">

**Author:** ![uniment](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uniment/32/24532_2.png) [@uniment](https://discourse.julialang.org/u/uniment)\
**Post date:** [November 26, 2022, 2:50am UTC](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue-rev-3/90836/1 "2022-11-26T02:50:46Z")

</div>

Third time’s (hopefully) the charm 😉

Okay so in contrast with my last two proposals ([1](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue/89654), [2](https://discourse.julialang.org/t/fixing-the-piping-chaining-partial-application-issue-rev-2/90408/)) this proposal is less preachy, and more “hey check out this crazy cool thing I did and please offer feedback.” Hopefully it’s good enough to be incorporated into the language. But play with it and let me know!

I got _really_ creative with method chaining over the past few days, and I’m really excited to share it.

# _Underscore Partial Application is Settled (imo)_

> **I think the problem of partial application is essentially already solved; it's just a matter of implementation.**
>
> Whether the Julia community has the will to accept it is a different matter, but that’s up to y’all! It boils down to these three steps:
> 
> 1. Accept that underscores will be used for partial application, and not for arbitrary lambdas.
> 2. Make a generalized functor for partial applicators, which underscore syntax lowers to.
> 3. Implement function composition as a default fallback when calling functions on these functors.
> 
> In the [last post](https://discourse.julialang.org/t/fixing-the-piping-chaining-partial-application-issue-rev-2/90408) I gave decent coverage of some ideas how this could be tackled, and I played with a [bunch of examples](https://discourse.julialang.org/t/fixing-the-piping-chaining-partial-application-issue-rev-2/90408/26) showing how function composition addresses the desire for multiple function calls.

So now I will focus on chaining as a separate matter, completely distinct from underscore syntax. We can get some really cool results if we focus specifically on chaining.

# _Single Chains_

Thanks to @christophE in [this post](https://discourse.julialang.org/t/partial-application-brackets-without-underscores/90565/4), I realized that dot `.` expressions with curly braces like `x.{f, g, h}` parse! And they already have all the behaviors I want, regarding a) precedence, b) being easy \*enough\* to type conveniently, and c) having convenient, unclaimed syntax for specifying un-executed chainlinks. And did I mention they already parse? It’s basically perfect.

Not only that, but because these expressions re-use the same parsing machinery as matrices, even 2-dimensional collections of expressions can be specified. What could that possibly be good for?? 🤔 I thought about it, and I went to town on it. Let’s see if you agree with me.

I put my implementation in a GitHub repo [MethodChains.jl](https://github.com/uniment/MethodChains.jl) so you can play along.

## The Standard Demos

I’m re-using the idea of having the local keyword `it` to refer to the last expression’s result, and of being the default argument for function calls. (See explanation for this decision [here](https://discourse.julialang.org/t/fixing-the-piping-chaining-partial-application-issue-rev-2/90408/4).) Each expression is assumed to be an expression of `it`, or to evaluate to a callable object which will be called on `it`.

Of course, it works just fine for the examples we’ve encountered so far.

_Basic Method Chains_

```julia
x.{f, g, h} ==  
x.{f; g; h} ==
x.{
    f
    g
    h
} ==
x.{f}.{g}.{h} ==  
h(g(f(x)))

```

Because the contents of `{}` already parse like the contents of `[]`, the same behaviors regarding commas, newlines, and semicolons carry over.

_Chains with Expressions of `it`_

```julia
x.{f, it+it^2, g} == g(let y=f(x); y+y^2 end)

```

_The Most Common Use for Chains are Short Expressions_

```julia
seq.{length} == length(seq)
obj.{first}.a == first(obj).a
map({it^2}, obj) == map(x->x^2, obj)
x.{cond ? f : g} == (cond ? f : g)(x)

```

_Examples from [Pipe.jl](https://github.com/oxinabox/Pipe.jl) Readme_

```julia
a.{b(it...)}
a.{b(it(1,2))}
a.{b(it[3])}
(2,4).{get_angle(it...)}

```

Longer Chains:

_Example from [Chain.jl](https://github.com/jkrumbiegel/Chain.jl) Readme_

```julia
df.{
    dropmissing
    filter(:id => >(6), it)
    groupby(it, :group)
    combine(it, :age => sum)
}

```

_Example from [DataPipes.jl](https://gitlab.com/aplavin/DataPipes.jl) Readme_

```julia
"a=1 b=2 c=3".{
    split,
    map({
        split(it, "=")
        (Symbol(it[1]) => parse(Int, it[2]))
    }, it),
    NamedTuple,
}

```

Also, if expressions in the chain are assignments, then they stay as-is and are not turned into assignments to `it`. This allows for declaring other variables local to the chain.

```julia
const avg = {len=it.{length}; sum(it)/len}
const stdev = {μ = it.{avg}, it.-μ, it.^2, avg, sqrt}

(1,2,3).{stdev} == 0.816496580927726

```

Note that chains and chainlinks are hard-scoped, so although they can access variables in their parent scope, they cannot change them.

Expressions which are not callable (e.g., `:tuple`, `:generator`, etc.) are simply assigned to `it` without attempting to call them or confirm that they’re expressions of `it`. This allows for things like this:

```julia
x = (1, 2)
x.{(a,b)=it, (; b,a)} == (b=2, a=1)

```

There are lots more demos in the [Readme](https://github.com/uniment/MethodChains.jl).

# _Multi-Chains_

This is where we enter the twilight zone. Early versions of Julia took inspiration from MatLab to use curly braces `{}` to define arrays, but the syntax became deprecated (yet the parser remains the same). As a result, now we have an unclaimed syntax for expressions spread across two dimensions.

What does it mean to have callable objects (and expressions of `it`) spread across two dimensions?

My thought is, they can represent multiple chains of execution, where progress down the vertical dimension represents the passage of time, and the horizontal dimension separates one chain of execution from another. The chains can intermittently interact by: a) having their values _copied_ across new chains, b) having their values _splatted_ across new chains, c) having their values _discarded_, or d) having their values _slurped_ into a collection. For (d), I reserve a new local keyword `them` which collects all the chains’ `it` values into a tuple.

Here’s an example showing splatting and slurping:

```julia
(a, b, c).{
    it...
    f g h
    g h f
    h f g
    it+3 it*2 it+1    
    them
}

```

The result is the tuple `(h(g(f(a)))+3, f(h(g(b)))*2, g(f(h(c)))+1)`. First `(a, b, c)` is splatted out across the columns, then each column performs its own operations, and the results are collected into a tuple `them` which is then returned.

I’ve implemented these rules:

1. A background chain, which maintains its local `it` and `them` values, is always present throughout the life of a multi-chain. Any local variables in the background chain are available to sub-chains (except for `it`, because each subchain declares its own local `it` which shadows the background chain’s).

2. Any time that more than one column is present, every column gets its own local subchain. The local subchain maintains its own `it` value, and can also have other local variables.  
– Note that sub-chains have hard local scopes, so they can’t alter the background chain’s local variables and they can’t alter each others’.

3. If the next row has a different number of columns than the last row, or if the last row has a splat `...` to spread an iterable object across the columns, or if the next row has a `them` to collect all the previous column’s `it` values, then _all chains are halted_ (and their local variables discarded), an inventory is taken of all of their `it` values (which are collected into a tuple `them`), and execution resumes with whatever new chains there are.  
– Fun note: the values can be collected and re-splatted with `them...`

4. When all sub-chains halt (which they must, in order for a single object to be returned), the background chain is resumed. By default its `it` value takes the left-most subchain’s `it` value, but `them` can be used to slurp up all the other subchains’ values into a tuple and return that instead.

Here are a couple examples:

_Compute Standard Deviation, Variance, and Maximum Absolute Deviation_

```julia
julia> @btime (0:10...,).{
           avg = {len=it.{length}, sum(it)/len}
           μ = it.{avg}
           it .- μ

         # stdev var mad
           it.^2 it.^2 abs.(it)
           avg avg maximum
           sqrt _ _
           them
       }
  1.000 ns (0 allocations: 0 bytes)
(3.1622776601683795, 10.0, 5.0)

```

The first two lines declared local variables for the background chain, then the third line set the background chain’s `it` value to something new. Then, when three columns appeared, the background chain’s `it` value was copied across, and each chain carried out its own instructions on it. For chains with no more commands, `_` is used to denote that the chain should do nothing, but not be halted or have its value discarded. On the last line, `them` collects all the chains’ values into a single tuple.

_FFT_

Can we implement functions which are graphically arranged to more closely match the underlying algorithm?

```julia
const toy_fft = {
    # setup
    Vector{ComplexF64}
    n = it.{length}
    if n == 2 return [it[1]+it[2]; it[1]-it[2]] else it end # base case
    W = exp(-2π*im/n)

    # butterfly
    it[1:2:end-1].{toy_fft} it[2:2:end].{toy_fft}
    _ it.*W.^(0:n÷2-1)
# ⋮ ⋱ ⋰ ⋮
                (x1,x2)=them
# ⋮ ⋰ ⋱ ⋮
    [x1.+x2 ; x1.-x2]::Vector{ComplexF64}
}

x = rand(1024)
X = x.{toy_fft}

```

Here, `toy_fft` is a recursive function that receives a single argument. The first line sets the collection type to a vector of complex numbers; the second line sets a local variable; the third line sets a base case return value; and the fourth line sets another local variable. From there, the background chain’s `it` value (which has so far been unchanged) is copied across two chains. Each chain takes its own slice of `it` (evens or odds), makes a recursive call on it, and the right chain applies phase-shifting [twiddle factors](https://en.wikipedia.org/wiki/Twiddle_factor). The line `(x1,x2)=them` collects the results and gives them convenient names, and then a [butterfly](https://en.wikipedia.org/wiki/Butterfly_diagram) is implemented on the final line and the results are concatenated for return.

Naturally, I tried calling this function on a tuple but it doesn’t work:

```julia
julia> (1, 2, 3, 4).{toy_fft}
ERROR: MethodError: no method matching Vector{ComplexF64}(::NTuple{4, Int64})

```

The keyword `it` sure comes in handy!

```julia
julia> (1, 2, 3, 4).{[it...], toy_fft}
4-element Vector{ComplexF64}:
                10.0 + 0.0im
                -2.0 + 2.0im
                -2.0 + 0.0im
 -1.9999999999999998 - 2.0im

```

# _Summary_

I’ve decided to make a go of using dot `.` syntax and curly braces `{}` for a method chaining syntax, and the result has been quite amazing and a lot of fun. The expressive flexibility of curly brace syntax enables chaining to generalize into entire function definitions with parallel tasks, while still making the simplest common things like accessing `obj.{first}.a` work a treat.

Please play with [MethodChains.jl](https://github.com/uniment/MethodChains.jl) and let me know your thoughts! I feel like this is a pretty good candidate for what curly brace syntax should be used for in the language, and help settle people’s desire to [relegate it to block delimiting](https://discourse.julialang.org/t/why-begin-end-instead-of-for-block-delimiter/89735/14).

I think the aspect I’m least settled on is how to manage the merging and splitting of chains. I haven’t worked it out, but I think you can implement arbitrary directed acyclic graphs of signal flows. However, any time a chain starts or halts, _all chains halt_ and an inventory is taken before starting up again. This makes it straightforward to reason about (esp. w.r.t. how to manage multiple expressions accessing `them`, or multiple splats in the same row), but there might be some use case I’m overlooking for which this is too constraining. Also, the default multi-chain alignment is left: so if a new chain begins (and if the previous row did not splat) the right-most value is copied over; and if a chain stops the right-most chain value is dropped. And in the end, when all the sub-chains stop, the left-most chain value is returned (unless they’re first collected with `them`).

The current behavior can be exemplified by the following implementation of a n=16 length FFT:

```julia
@mc const fft2 = { 
    (x1,x2)=it
    (x1+x2, x1-x2)
}

@mc const fft4 = {
    it[1:2:3].{fft2}... it[2:2:4].{fft2}...
    _ _ it*1 -it*im;
    (x1,x2)=(them[1:2], them[3:4])
    (x1.+x2..., x1.-x2...)
}

@mc const fft8 = {
    ∠ = √-im
    it[1:2:7].{fft4}... it[2:2:8].{fft4}...
    _ _ _ _ it*1 it*∠ -it*im it*∠^3
    (x1,x2)=(them[1:4], them[5:8])
    (x1.+x2..., x1.-x2...) 
}

@mc const fft16 = {
    ∠ = √-im; ∠∠ = √∠
    it[1:2:15].{fft8}... it[2:2:16].{fft8}...
    _ _ _ _ _ _ _ _ it*1 it*∠∠ it*∠ it*∠∠^3 -it*im it*∠∠^5 it*∠^3 it*∠∠^7
    (x1,x2)=(them[1:8], them[9:16])
    (x1.+x2..., x1.-x2...)
}

```

In the definition of `fft16`, the result of calling `fft8` on an even or odd slice of `it` is splatted across the next row; the left eight are left unchanged, while the right eight are twiddled by various fractions of a half complex unit circle. Then, slices of `them` are collected and assigned to `x1` and `x2` for easy broadcasted arithmetic, before being re-splatted into a tuple.

All that to say, the current arrangement seems to have its merits and is easy enough to reason about; I feel like it’s pretty good. But I don’t know if it’s the best possible option or not. Your thoughts are very welcome.

GLHF!

---

_[View the full topic](https://discourse.julialang.org/t/fixing-the-piping-chaining-issue-rev-3/90836)._