Fixing the Piping/Chaining Issue (Rev 3)

uniment
Sat, 26 Nov 2022 02:50:46 +0000
Internals & Design

Third time’s (hopefully) the charm

Okay so in contrast with my last two proposals (1, 2) this proposal is less preachy, and more “hey check out this crazy cool thing I did and please offer feedback.” Hopefully it’s good enough to be incorporated into the language. But play with it and let me know!

I got really creative with method chaining over the past few days, and I’m really excited to share it.

Underscore Partial Application is Settled (imo)
I think the problem of partial application is essentially already solved; it's just a matter of implementation. (click for more details)
So now I will focus on chaining as a separate matter, completely distinct from underscore syntax. We can get some really cool results if we focus specifically on chaining.

Single Chains

Thanks to @christophE in this post, I realized that dot . expressions with curly braces like x.{f, g, h} parse! And they already have all the behaviors I want, regarding a) precedence, b) being easy *enough* to type conveniently, and c) having convenient, unclaimed syntax for specifying un-executed chainlinks. And did I mention they already parse? It’s basically perfect.

Not only that, but because these expressions re-use the same parsing machinery as matrices, even 2-dimensional collections of expressions can be specified. What could that possibly be good for?? I thought about it, and I went to town on it. Let’s see if you agree with me.

I put my implementation in a GitHub repo MethodChains.jl so you can play along.

The Standard Demos

I’m re-using the idea of having the local keyword it to refer to the last expression’s result, and of being the default argument for function calls. (See explanation for this decision here.) Each expression is assumed to be an expression of it, or to evaluate to a callable object which will be called on it.

Of course, it works just fine for the examples we’ve encountered so far.

Basic Method Chains

x.{f, g, h}    ==
x.{f; g; h}    ==
x.{
    f
    g
    h
} ==
x.{f}.{g}.{h}  ==
h(g(f(x)))

Because the contents of {} already parse like the contents of [], the same behaviors regarding commas, newlines, and semicolons carry over.

Chains with Expressions of it

x.{f, it+it^2, g} == g(let y=f(x); y+y^2 end)

The Most Common Use for Chains are Short Expressions

seq.{length}      ==  length(seq)
obj.{first}.a     ==  first(obj).a
map({it^2}, obj)  ==  map(x->x^2, obj)
x.{cond ? f : g}  ==  (cond ? f : g)(x)

Examples from Pipe.jl Readme

a.{b(it...)}
a.{b(it(1,2))}
a.{b(it[3])}
(2,4).{get_angle(it...)}

Longer Chains:

Example from Chain.jl Readme

df.{
    dropmissing
    filter(:id => >(6), it)
    groupby(it, :group)
    combine(it, :age => sum)
}

Example from DataPipes.jl Readme

"a=1 b=2 c=3".{
    split,
    map({
        split(it, "=")
        (Symbol(it[1]) => parse(Int, it[2]))
    }, it),
    NamedTuple,
}

Also, if expressions in the chain are assignments, then they stay as-is and are not turned into assignments to it. This allows for declaring other variables local to the chain.

const avg = {len=it.{length}; sum(it)/len}
const stdev = {μ = it.{avg}, it.-μ, it.^2, avg, sqrt}

(1,2,3).{stdev} == 0.816496580927726

Note that chains and chainlinks are hard-scoped, so although they can access variables in their parent scope, they cannot change them.

Expressions which are not callable (e.g., :tuple, :generator, etc.) are simply assigned to it without attempting to call them or confirm that they’re expressions of it. This allows for things like this:

x = (1, 2)
x.{(a,b)=it, (; b,a)} == (b=2, a=1)

There are lots more demos in the Readme.

Multi-Chains

This is where we enter the twilight zone. Early versions of Julia took inspiration from MatLab to use curly braces {} to define arrays, but the syntax became deprecated (yet the parser remains the same). As a result, now we have an unclaimed syntax for expressions spread across two dimensions.

What does it mean to have callable objects (and expressions of it) spread across two dimensions?

My thought is, they can represent multiple chains of execution, where progress down the vertical dimension represents the passage of time, and the horizontal dimension separates one chain of execution from another. The chains can intermittently interact by: a) having their values copied across new chains, b) having their values splatted across new chains, c) having their values discarded, or d) having their values slurped into a collection. For (d), I reserve a new local keyword them which collects all the chains’ it values into a tuple.

Here’s an example showing splatting and slurping:

(a, b, c).{
    it...
    f       g       h
    g       h       f
    h       f       g
    it+3    it*2    it+1
    them
}

The result is the tuple (h(g(f(a)))+3, f(h(g(b)))*2, g(f(h(c)))+1). First (a, b, c) is splatted out across the columns, then each column performs its own operations, and the results are collected into a tuple them which is then returned.

I’ve implemented these rules:

1. A background chain, which maintains its local it and them values, is always present throughout the life of a multi-chain. Any local variables in the background chain are available to sub-chains (except for it, because each subchain declares its own local it which shadows the background chain’s).

2. Any time that more than one column is present, every column gets its own local subchain. The local subchain maintains its own it value, and can also have other local variables.
– Note that sub-chains have hard local scopes, so they can’t alter the background chain’s local variables and they can’t alter each others’.

3. If the next row has a different number of columns than the last row, or if the last row has a splat ... to spread an iterable object across the columns, or if the next row has a them to collect all the previous column’s it values, then all chains are halted (and their local variables discarded), an inventory is taken of all of their it values (which are collected into a tuple them), and execution resumes with whatever new chains there are.
– Fun note: the values can be collected and re-splatted with them...

4. When all sub-chains halt (which they must, in order for a single object to be returned), the background chain is resumed. By default its it value takes the left-most subchain’s it value, but them can be used to slurp up all the other subchains’ values into a tuple and return that instead.

Here are a couple examples:

Compute Standard Deviation, Variance, and Maximum Absolute Deviation

julia> @btime (0:10...,).{
           avg = {len=it.{length}, sum(it)/len}
           μ = it.{avg}
           it .- μ

         # stdev     var      mad
           it.^2     it.^2    abs.(it)
           avg       avg      maximum
           sqrt      _        _
           them
       }
  1.000 ns (0 allocations: 0 bytes)
(3.1622776601683795, 10.0, 5.0)

The first two lines declared local variables for the background chain, then the third line set the background chain’s it value to something new. Then, when three columns appeared, the background chain’s it value was copied across, and each chain carried out its own instructions on it. For chains with no more commands, _ is used to denote that the chain should do nothing, but not be halted or have its value discarded. On the last line, them collects all the chains’ values into a single tuple.

FFT

Can we implement functions which are graphically arranged to more closely match the underlying algorithm?

const toy_fft = {
    # setup
    Vector{ComplexF64}
    n = it.{length}
    if n == 2 return [it[1]+it[2]; it[1]-it[2]] else it end # base case
    W = exp(-2π*im/n)

    # butterfly
    it[1:2:end-1].{toy_fft}   it[2:2:end].{toy_fft}
    _                         it.*W.^(0:n÷2-1)
#   ⋮        ⋱                ⋰         ⋮
                (x1,x2)=them
#   ⋮        ⋰                ⋱         ⋮
    [x1.+x2          ;            x1.-x2]::Vector{ComplexF64}
}

x = rand(1024)
X = x.{toy_fft}

Here, toy_fft is a recursive function that receives a single argument. The first line sets the collection type to a vector of complex numbers; the second line sets a local variable; the third line sets a base case return value; and the fourth line sets another local variable. From there, the background chain’s it value (which has so far been unchanged) is copied across two chains. Each chain takes its own slice of it (evens or odds), makes a recursive call on it, and the right chain applies phase-shifting twiddle factors. The line (x1,x2)=them collects the results and gives them convenient names, and then a butterfly is implemented on the final line and the results are concatenated for return.

Naturally, I tried calling this function on a tuple but it doesn’t work:

julia> (1, 2, 3, 4).{toy_fft}
ERROR: MethodError: no method matching Vector{ComplexF64}(::NTuple{4, Int64})

The keyword it sure comes in handy!

julia> (1, 2, 3, 4).{[it...], toy_fft}
4-element Vector{ComplexF64}:
                10.0 + 0.0im
                -2.0 + 2.0im
                -2.0 + 0.0im
 -1.9999999999999998 - 2.0im

Summary

I’ve decided to make a go of using dot . syntax and curly braces {} for a method chaining syntax, and the result has been quite amazing and a lot of fun. The expressive flexibility of curly brace syntax enables chaining to generalize into entire function definitions with parallel tasks, while still making the simplest common things like accessing obj.{first}.a work a treat.

Please play with MethodChains.jl and let me know your thoughts! I feel like this is a pretty good candidate for what curly brace syntax should be used for in the language, and help settle people’s desire to relegate it to block delimiting.

I think the aspect I’m least settled on is how to manage the merging and splitting of chains. I haven’t worked it out, but I think you can implement arbitrary directed acyclic graphs of signal flows. However, any time a chain starts or halts, all chains halt and an inventory is taken before starting up again. This makes it straightforward to reason about (esp. w.r.t. how to manage multiple expressions accessing them, or multiple splats in the same row), but there might be some use case I’m overlooking for which this is too constraining. Also, the default multi-chain alignment is left: so if a new chain begins (and if the previous row did not splat) the right-most value is copied over; and if a chain stops the right-most chain value is dropped. And in the end, when all the sub-chains stop, the left-most chain value is returned (unless they’re first collected with them).

The current behavior can be exemplified by the following implementation of a n=16 length FFT:

@mc const fft2 = {
    (x1,x2)=it
    (x1+x2, x1-x2)
}

@mc const fft4 = {
    it[1:2:3].{fft2}...   it[2:2:4].{fft2}...
    _       _       it*1        -it*im;
    (x1,x2)=(them[1:2], them[3:4])
    (x1.+x2..., x1.-x2...)
}

@mc const fft8 = {
    ∠ = √-im
    it[1:2:7].{fft4}...   it[2:2:8].{fft4}...
    _   _   _   _   it*1    it*∠  -it*im  it*∠^3
    (x1,x2)=(them[1:4], them[5:8])
    (x1.+x2..., x1.-x2...)
}

@mc const fft16 = {
    ∠ = √-im; ∠∠ = √∠
    it[1:2:15].{fft8}...  it[2:2:16].{fft8}...
    _   _   _   _   _   _   _   _   it*1   it*∠∠   it*∠   it*∠∠^3   -it*im   it*∠∠^5   it*∠^3   it*∠∠^7
    (x1,x2)=(them[1:8], them[9:16])
    (x1.+x2..., x1.-x2...)
}

In the definition of fft16, the result of calling fft8 on an even or odd slice of it is splatted across the next row; the left eight are left unchanged, while the right eight are twiddled by various fractions of a half complex unit circle. Then, slices of them are collected and assigned to x1 and x2 for easy broadcasted arithmetic, before being re-splatted into a tuple.

All that to say, the current arrangement seems to have its merits and is easy enough to reason about; I feel like it’s pretty good. But I don’t know if it’s the best possible option or not. Your thoughts are very welcome.

GLHF!

90 posts - 26 participants