Third time’s (hopefully) the charm 
Okay so in contrast with my last two proposals (1, 2) this proposal is less preachy, and more “hey check out this crazy cool thing I did and please offer feedback.” Hopefully it’s good enough to be incorporated into the language. But play with it and let me know!
I got really creative with method chaining over the past few days, and I’m really excited to share it.
Underscore Partial Application is Settled (imo)
I think the problem of partial application is essentially already solved; it's just a matter of implementation.
Whether the Julia community has the will to accept it is a different matter, but that’s up to y’all! It boils down to these three steps:
- Accept that underscores will be used for partial application, and not for arbitrary lambdas.
- Make a generalized functor for partial applicators, which underscore syntax lowers to.
- Implement function composition as a default fallback when calling functions on these functors.
In the last post I gave decent coverage of some ideas how this could be tackled, and I played with a bunch of examples showing how function composition addresses the desire for multiple function calls.
So now I will focus on chaining as a separate matter, completely distinct from underscore syntax. We can get some really cool results if we focus specifically on chaining.
Single Chains
Thanks to @christophE in this post, I realized that dot . expressions with curly braces like x.{f, g, h} parse! And they already have all the behaviors I want, regarding a) precedence, b) being easy *enough* to type conveniently, and c) having convenient, unclaimed syntax for specifying un-executed chainlinks. And did I mention they already parse? It’s basically perfect.
Not only that, but because these expressions re-use the same parsing machinery as matrices, even 2-dimensional collections of expressions can be specified. What could that possibly be good for?? 
I thought about it, and I went to town on it. Let’s see if you agree with me.
I put my implementation in a GitHub repo MethodChains.jl so you can play along.
The Standard Demos
I’m re-using the idea of having the local keyword it to refer to the last expression’s result, and of being the default argument for function calls. (See explanation for this decision here.) Each expression is assumed to be an expression of it, or to evaluate to a callable object which will be called on it.
Of course, it works just fine for the examples we’ve encountered so far.
Basic Method Chains
x.{f, g, h} ==
x.{f; g; h} ==
x.{
f
g
h
} ==
x.{f}.{g}.{h} ==
h(g(f(x)))
Because the contents of {} already parse like the contents of [], the same behaviors regarding commas, newlines, and semicolons carry over.
Chains with Expressions of it
x.{f, it+it^2, g} == g(let y=f(x); y+y^2 end)
The Most Common Use for Chains are Short Expressions
seq.{length} == length(seq)
obj.{first}.a == first(obj).a
map({it^2}, obj) == map(x->x^2, obj)
x.{cond ? f : g} == (cond ? f : g)(x)
Examples from Pipe.jl Readme
a.{b(it...)}
a.{b(it(1,2))}
a.{b(it[3])}
(2,4).{get_angle(it...)}
Longer Chains:
Example from Chain.jl Readme
df.{
dropmissing
filter(:id => >(6), it)
groupby(it, :group)
combine(it, :age => sum)
}
Example from DataPipes.jl Readme
"a=1 b=2 c=3".{
split,
map({
split(it, "=")
(Symbol(it[1]) => parse(Int, it[2]))
}, it),
NamedTuple,
}
Also, if expressions in the chain are assignments, then they stay as-is and are not turned into assignments to it. This allows for declaring other variables local to the chain.
const avg = {len=it.{length}; sum(it)/len}
const stdev = {μ = it.{avg}, it.-μ, it.^2, avg, sqrt}
(1,2,3).{stdev} == 0.816496580927726
Note that chains and chainlinks are hard-scoped, so although they can access variables in their parent scope, they cannot change them.
Expressions which are not callable (e.g., :tuple, :generator, etc.) are simply assigned to it without attempting to call them or confirm that they’re expressions of it. This allows for things like this:
x = (1, 2)
x.{(a,b)=it, (; b,a)} == (b=2, a=1)
There are lots more demos in the Readme.
Multi-Chains
This is where we enter the twilight zone. Early versions of Julia took inspiration from MatLab to use curly braces {} to define arrays, but the syntax became deprecated (yet the parser remains the same). As a result, now we have an unclaimed syntax for expressions spread across two dimensions.
What does it mean to have callable objects (and expressions of it) spread across two dimensions?
My thought is, they can represent multiple chains of execution, where progress down the vertical dimension represents the passage of time, and the horizontal dimension separates one chain of execution from another. The chains can intermittently interact by: a) having their values copied across new chains, b) having their values splatted across new chains, c) having their values discarded, or d) having their values slurped into a collection. For (d), I reserve a new local keyword them which collects all the chains’ it values into a tuple.
Here’s an example showing splatting and slurping:
(a, b, c).{
it...
f g h
g h f
h f g
it+3 it*2 it+1
them
}
The result is the tuple (h(g(f(a)))+3, f(h(g(b)))*2, g(f(h(c)))+1). First (a, b, c) is splatted out across the columns, then each column performs its own operations, and the results are collected into a tuple them which is then returned.
I’ve implemented these rules:
-
A background chain, which maintains its local
itandthemvalues, is always present throughout the life of a multi-chain. Any local variables in the background chain are available to sub-chains (except forit, because each subchain declares its own localitwhich shadows the background chain’s). -
Any time that more than one column is present, every column gets its own local subchain. The local subchain maintains its own
itvalue, and can also have other local variables.
– Note that sub-chains have hard local scopes, so they can’t alter the background chain’s local variables and they can’t alter each others’. -
If the next row has a different number of columns than the last row, or if the last row has a splat
...to spread an iterable object across the columns, or if the next row has athemto collect all the previous column’sitvalues, then all chains are halted (and their local variables discarded), an inventory is taken of all of theiritvalues (which are collected into a tuplethem), and execution resumes with whatever new chains there are.
– Fun note: the values can be collected and re-splatted withthem... -
When all sub-chains halt (which they must, in order for a single object to be returned), the background chain is resumed. By default its
itvalue takes the left-most subchain’sitvalue, butthemcan be used to slurp up all the other subchains’ values into a tuple and return that instead.
Here are a couple examples:
Compute Standard Deviation, Variance, and Maximum Absolute Deviation
julia> @btime (0:10...,).{
avg = {len=it.{length}, sum(it)/len}
μ = it.{avg}
it .- μ
# stdev var mad
it.^2 it.^2 abs.(it)
avg avg maximum
sqrt _ _
them
}
1.000 ns (0 allocations: 0 bytes)
(3.1622776601683795, 10.0, 5.0)
The first two lines declared local variables for the background chain, then the third line set the background chain’s it value to something new. Then, when three columns appeared, the background chain’s it value was copied across, and each chain carried out its own instructions on it. For chains with no more commands, _ is used to denote that the chain should do nothing, but not be halted or have its value discarded. On the last line, them collects all the chains’ values into a single tuple.
FFT
Can we implement functions which are graphically arranged to more closely match the underlying algorithm?
const toy_fft = {
# setup
Vector{ComplexF64}
n = it.{length}
if n == 2 return [it[1]+it[2]; it[1]-it[2]] else it end # base case
W = exp(-2π*im/n)
# butterfly
it[1:2:end-1].{toy_fft} it[2:2:end].{toy_fft}
_ it.*W.^(0:n÷2-1)
# ⋮ ⋱ ⋰ ⋮
(x1,x2)=them
# ⋮ ⋰ ⋱ ⋮
[x1.+x2 ; x1.-x2]::Vector{ComplexF64}
}
x = rand(1024)
X = x.{toy_fft}
Here, toy_fft is a recursive function that receives a single argument. The first line sets the collection type to a vector of complex numbers; the second line sets a local variable; the third line sets a base case return value; and the fourth line sets another local variable. From there, the background chain’s it value (which has so far been unchanged) is copied across two chains. Each chain takes its own slice of it (evens or odds), makes a recursive call on it, and the right chain applies phase-shifting twiddle factors. The line (x1,x2)=them collects the results and gives them convenient names, and then a butterfly is implemented on the final line and the results are concatenated for return.
Naturally, I tried calling this function on a tuple but it doesn’t work:
julia> (1, 2, 3, 4).{toy_fft}
ERROR: MethodError: no method matching Vector{ComplexF64}(::NTuple{4, Int64})
The keyword it sure comes in handy!
julia> (1, 2, 3, 4).{[it...], toy_fft}
4-element Vector{ComplexF64}:
10.0 + 0.0im
-2.0 + 2.0im
-2.0 + 0.0im
-1.9999999999999998 - 2.0im
Summary
I’ve decided to make a go of using dot . syntax and curly braces {} for a method chaining syntax, and the result has been quite amazing and a lot of fun. The expressive flexibility of curly brace syntax enables chaining to generalize into entire function definitions with parallel tasks, while still making the simplest common things like accessing obj.{first}.a work a treat.
Please play with MethodChains.jl and let me know your thoughts! I feel like this is a pretty good candidate for what curly brace syntax should be used for in the language, and help settle people’s desire to relegate it to block delimiting.
I think the aspect I’m least settled on is how to manage the merging and splitting of chains. I haven’t worked it out, but I think you can implement arbitrary directed acyclic graphs of signal flows. However, any time a chain starts or halts, all chains halt and an inventory is taken before starting up again. This makes it straightforward to reason about (esp. w.r.t. how to manage multiple expressions accessing them, or multiple splats in the same row), but there might be some use case I’m overlooking for which this is too constraining. Also, the default multi-chain alignment is left: so if a new chain begins (and if the previous row did not splat) the right-most value is copied over; and if a chain stops the right-most chain value is dropped. And in the end, when all the sub-chains stop, the left-most chain value is returned (unless they’re first collected with them).
The current behavior can be exemplified by the following implementation of a n=16 length FFT:
@mc const fft2 = {
(x1,x2)=it
(x1+x2, x1-x2)
}
@mc const fft4 = {
it[1:2:3].{fft2}... it[2:2:4].{fft2}...
_ _ it*1 -it*im;
(x1,x2)=(them[1:2], them[3:4])
(x1.+x2..., x1.-x2...)
}
@mc const fft8 = {
∠ = √-im
it[1:2:7].{fft4}... it[2:2:8].{fft4}...
_ _ _ _ it*1 it*∠ -it*im it*∠^3
(x1,x2)=(them[1:4], them[5:8])
(x1.+x2..., x1.-x2...)
}
@mc const fft16 = {
∠ = √-im; ∠∠ = √∠
it[1:2:15].{fft8}... it[2:2:16].{fft8}...
_ _ _ _ _ _ _ _ it*1 it*∠∠ it*∠ it*∠∠^3 -it*im it*∠∠^5 it*∠^3 it*∠∠^7
(x1,x2)=(them[1:8], them[9:16])
(x1.+x2..., x1.-x2...)
}
In the definition of fft16, the result of calling fft8 on an even or odd slice of it is splatted across the next row; the left eight are left unchanged, while the right eight are twiddled by various fractions of a half complex unit circle. Then, slices of them are collected and assigned to x1 and x2 for easy broadcasted arithmetic, before being re-splatted into a tuple.
All that to say, the current arrangement seems to have its merits and is easy enough to reason about; I feel like it’s pretty good. But I don’t know if it’s the best possible option or not. Your thoughts are very welcome.
GLHF!
