It’s nice that there are more attentions to transducers! So, I wanted to write some comments somewhere but I didn’t want to get it lost in the unrecoverable slack history. Discourse seems to be a good place for this, although the post itself does not mention Julia at all
Anyway, first, let me comment on the limitations of transducers as mentioned in the post. I think there are a few misconceptions (some common ones, I think) partially due to mixing the transducer as an idea and transducer as implemented in Clojure:
For example, there is nothing stopping you from implementing stateful transducers using pure functions. That’s done in C++ and that’s how I implemented in Julia.
I think the double-wrap problem is just a problem in Clojure implementation. IIRC, Rich Hickey commented somewhere that he wished he never have done that. (So I avoided this in my implementation.)
foldlis not hard. It’s actually much easier than implementing
iterate. With a bit of syntax sugar, it’s as easy as Python’s
yield. See: https://github.com/JuliaFolds/GeneratorsX.jl
One thing I wish that was mentioned as a limitation of
foldl approach (≈ “transducer approach”) is that it does not compose “horizontally” (the terminology I just made up) in the sense you can’t write
zip with it. OTOH, it’s very easy to write
iterate. (Generally, some hard things in
foldl are easy in
iterate and vice versa.)
It’d be very interesting to see how
zip would be handled by the effect handler framework.
In general, I’m skeptical if you can do something essentially different from
foldl in this space. They are pretty minimal and they are the dual of each other (see Catamorphism - Wikipedia). To emphasize this, let me comment on the first sentence (emphasis by me):
Clojure has introduced a very interesting idea called ‘transducers’, which decouple sequence transformations (mapping, filtering etc) from sequence implementations like vectors, lazy lists and channels.
I actually wouldn’t say “decoupling sequence transformations” as the defining property of transducers. The transducers transform reducing step functions. It’s also possible to define what I call iterator transforms that transform iterators (which is just a curried form of what we have in
itr -> Iterators.filter(p, itr) is an iterator transform). So, data-independent composition of the transformations can happen at “iterator side.”
My impression is that the effect handler is essentially a syntax sugar for defining
iterate (or in some cases an iterator transform). I do agree this direction is very powerful, though. This is because it’s just so hard to create state machine by hand and it does not perform well (at least with the current
julia/LLVM compiler). That’s why I’m exploring something similar in GeneratorsX.jl (based on Mike’s IRTools.jl!).