# ANN: Transducers.jl, efficient and composable algorithms for map- and reduce-like operations

**URL:** <https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159>\
**Category:** Package Announcements\
**Created:** [January 1, 2019, 3:30am UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159 "2019-01-01T03:30:43Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 1, 2019, 3:30am UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/1 "2019-01-01T03:30:43Z")

</div>

Here is a quote from the [README](https://github.com/tkf/Transducers.jl):

> Transducers.jl provides composable algorithms on sequence of inputs.  
> They are called _[transducers](https://clojure.org/reference/transducers)_, first introduced in Clojure language  
> by Rich Hickey.
> 
> Using transducers is quite straightforward, especially if you already  
> know similar concepts in iterator libraries:
> 
> ```julia
> using Transducers
> xf = Partition(7) |> Filter(x -> prod(x) % 11 == 0) |> Cat() |> Scan(+)
> mapfoldl(xf, +, 1:40)
> 
> ```
> 
> However, the formalization of the transducers is quite different from  
> iterators and resulting in a better performance for complex  
> compositions.
> 
> See more in the [documentation](https://tkf.github.io/Transducers.jl/latest).
> 
> #### Installation
> 
> ```julia
> ]add https://github.com/tkf/Transducers.jl
> 
> ```

There are more [examples](https://tkf.github.io/Transducers.jl/latest/#Examples-1) in the documentation (see also [reference manual](https://tkf.github.io/Transducers.jl/latest/manual/#Transducers-1)). I also discussed [difference to iterators in the documentation](https://tkf.github.io/Transducers.jl/latest/#Difference-to-iterators-1).

Some (technical) highlights:

- It turned out the composed/lowered code is quite compiler-friendly. For example, `julia` generates [SIMD instructions for `Filter(...) -> Map(...)`](https://github.com/tkf/Transducers.jl/blob/2fdc400b6d1a9ad6d5970ffd3c8d2c1d88e5e0b5/test/__test_ir.jl#L28-L33) without me explicitly coding for SIMD.
- “Nested loop” construct like `Cat` (concatenation) is quite easy to write (and efficient). No state juggling as in iterators. It’s a good fit for the split-apply-combine pattern.
- This also applies to the container types. If you have a container type that needs nested for loops, it becomes non-trivial to write `Base.iterate` while supporting transducers is quite straightforward. See also: [How to make your data type reducible](https://tkf.github.io/Transducers.jl/latest/examples/reducibles/).
- A subset of transducers support parallelism (based on `Base.Threads`). Here is [an example of splitting a string into words and counting them in parallel](https://tkf.github.io/Transducers.jl/latest/examples/words/), partially based on [Guy Steele’s 2009 ICFP talk](https://vimeo.com/6624203) (which is a good talk to watch, by the way).

Implementing transducers in a Julia-friendly way was an interesting holiday project. It even makes me wonder why it’s not as main stream as iterators (outside Clojure world?). I know there is a [C++ library](https://github.com/Ableton/atria) which is actually where I get the idea of type-stability consideration. Anyway, if you have any thoughts on this, I’d like to hear.

Wish you a happy new year!

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [January 1, 2019, 3:08pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/2 "2019-01-01T15:08:50Z")

</div>

This is very cool and really impressive for a “holiday project”! I’ve been meaning to spend some time looking at clojure’s transducers ever since Rich Hickey first introduced them and not gotten around to it. Now I guess there’s no excuse since there’s a Julia implementation 😁

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 1, 2019, 7:11pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/3 "2019-01-01T19:11:10Z")

</div>

Thanks! It’s good to know that transducer was on your radar 🙂

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [January 1, 2019, 8:20pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/4 "2019-01-01T20:20:47Z")

</div>

Wow, this is really cool! It feels very Julian.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [January 1, 2019, 9:16pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/5 "2019-01-01T21:16:31Z")

</div>

Sorry to ask for numbers, but I’m curious as to the advantages of Transducers, any bench-marks showing how much better this is than iterators?

---

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [January 1, 2019, 9:44pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/6 "2019-01-01T21:44:24Z")

</div>

To me the selling point of `Transducers` are composability and generality. I would not say that they are “faster” then iterators. They are not faster than a hand tuned for loop. But they avoid materializing intermediate results and thus they are faster then high level iterator code. Here is the example from @tkf’s SIMD test:

Edit: This snipped is misleading. I benchmark apples against oranges. See [@tkf’s comment below](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/8).

```julia
julia> using Transducers, BenchmarkTools

julia> xs = randn(1000); ys = similar(xs);

julia> xf = Filter(x -> -0.5 < x < 0.5) |> Map(x -> 2x)
Filter(Main.λ❓) |>
    Map(Main.λ❓)

julia> @benchmark map!(xf,ys,xs)
BenchmarkTools.Trial: 
  memory estimate: 0 bytes
  allocs estimate: 0
  --------------
  minimum time: 953.750 ns (0.00% GC)
  median time: 965.950 ns (0.00% GC)
  mean time: 977.053 ns (0.00% GC)
  maximum time: 1.890 μs (0.00% GC)
  --------------
  samples: 10000
  evals/sample: 20

julia> @benchmark map!(x -> 2x, ys, filter(x -> -0.5 < x < 0.5, xs))
BenchmarkTools.Trial: 
  memory estimate: 8.39 KiB
  allocs estimate: 9
  --------------
  minimum time: 4.883 μs (0.00% GC)
  median time: 5.311 μs (0.00% GC)
  mean time: 7.015 μs (16.79% GC)
  maximum time: 6.901 ms (99.78% GC)
  --------------
  samples: 10000
  evals/sample: 6

```

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 1, 2019, 9:44pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/7 "2019-01-01T21:44:54Z")

</div>

Good point. Here is an example that is very “hostile” to iterator:

```julia
julia> xs = 1:2^15
1:32768

julia> @btime foldl(+, Iterators.flatten(1:x for x in xs))
  200.448 ms (10 allocations: 352 bytes)
5864598896640

julia> @btime mapfoldl(Map(x -> 1:x) |> Cat(), +, xs)
  45.532 μs (18 allocations: 880 bytes)
5864598896640

```

Piggybacking on [Skipnothing is missing](https://discourse.julialang.org/t/skipnothing-is-missing/19155), here is a more reasonable case for comparison:

```julia
julia> xs = [abs(x) > 1 ? nothing : x for x in randn(2^20)];

julia> @btime foldl(+, Iterators.filter(!isnothing, xs); init=0.0)
  4.084 ms (4 allocations: 80 bytes)
779.0559461186579

julia> @btime mapfoldl(Filter(!isnothing), +, xs, init=0.0)
  2.913 ms (5 allocations: 80 bytes)
779.0559461186579

```

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 1, 2019, 9:50pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/8 "2019-01-01T21:50:35Z")

</div>

Thanks for bringing up that example. Let me note that the semantics of `map!` for transducers and iterators are bit different. Transducer `Filter` skips the destination elements as well while the iterator version “compress” all the output in a contiguous chunk. A function that does a similar thing in Transducers.jl is `copy!` (we may need `copyto!` which could be more efficient).

---

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [January 1, 2019, 9:50pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/9 "2019-01-01T21:50:57Z")

</div>

@tkf cool! Do you have an explanation, why the skip nothing example if faster using `Transducers`?

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 1, 2019, 10:27pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/10 "2019-01-01T22:27:55Z")

</div>

I need to look at IR to be sure but some guesses:

- `foldl` in `Base` actually does a lot more than what I’ve implemented in Transducers.jl, like detecting correct `zero` even when `eltype(xs)` is `Union{Nothing, Float64}`. This may give it some overhead to the iterator version (although I tried to be slightly fair by passing an `init`).

- Maybe I put too many `@inline`s 🙂

- But it could also be a real effect; as I wrote in [difference to iterators](https://tkf.github.io/Transducers.jl/latest/#Difference-to-iterators-1), the code “generated” by the iterator is a nested `while` loop with conditional break. Maybe Julia compiler can optimize structured `for` loop more easily? Although this means in the future there would probably be no difference in performance.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 2, 2019, 6:19pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/11 "2019-01-02T18:19:50Z")

</div>

I find something very bizarre: Transducer is actually faster than “equivalent” manual loop. I took `sum_nonmissing` from [First-Class Statistical Missing Values Support in Julia 0.7](https://julialang.org/blog/2018/06/missing) which noted:

> replacing `x !== missing` by `ismissing(x)` in `sum_nonmissing` currently leads to a large performance drop

Indeed, `sum_isnotmissing` below is much slower than `sum_nonmissing` (as of 1.2.0-DEV.63).

Keno explained in [the issue comment linked from the blog](https://github.com/JuliaLang/julia/issues/27681) this is because:

> `===` is special in inference and inference happens before inlining, so even if they’re the same after inlining, Inference doesn’t know that.

_However_, using `Filter(!ismissing)` is just as fast as using `x !== missing` in the manual loop. If I look at `@code_warntype` output, I can see that output type is correctly inferred.

I’m not entirely sure what’s going on, but I’m guessing that transducers are compiler-friendly because they aggressively put function boundaries which may help inference.

```julia
using Transducers
using BenchmarkTools

# Taken from: https://julialang.org/blog/2018/06/missing
function sum_nonmissing(X::AbstractArray)
    s = zero(eltype(X))
    @inbounds @simd for x in X
        if x !== missing
            s += x
        end
    end
    s
end

function sum_isnotmissing(X::AbstractArray)
    s = zero(eltype(X))
    @inbounds @simd for x in X
        if !ismissing(x)
            s += x
        end
    end
    s
end

# Benchmark:

n = 2^10
xs = [abs(x) > 0.5 ? missing : x for x in randn(n)]

@btime sum_isnotmissing($xs)
# 2.908 μs (0 allocations: 0 bytes)
@btime sum_nonmissing($xs)
# 772.896 ns (0 allocations: 0 bytes)
@btime mapfoldl($(Filter(!ismissing)), +, $xs, init=0.0)
# 784.798 ns (0 allocations: 0 bytes)
VERSION
# v"1.2.0-DEV.63" (similar result with 1.0)

```

---

<div class="post-metadata">

**Author:** ![andyferris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andyferris/32/235_2.png) [@andyferris](https://discourse.julialang.org/u/andyferris)\
**Post date:** [January 2, 2019, 9:00pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/12 "2019-01-02T21:00:20Z")

</div>

OK this is very cool, @tkf!

Just one question - it seems that `Partition`, `Filter`, `Cat`, `Scan`, etc behave like `Function`s - why is that they are piped into each other via `|>` (evaluate function with data operator) instead of composed with `∘` (function followed by another function operator)? Is it just to get the order of operations looking correct? (I’ve often wanted to have something like `∘` but writing the functions in the other order).

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 2, 2019, 10:54pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/13 "2019-01-02T22:54:38Z")

</div>

Thanks for the comment!

> [@andyferris](#):
>
> it seems that `Partition` , `Filter` , `Cat` , `Scan` , etc behave like `Function` s - why is that they are piped into each other via `|>` (evaluate function with data operator) instead of composed with `∘` (function followed by another function operator)?

First, just to clarify, I defined `|>(::Transducer, ::Transducer)` directly without making `Transducer` a callable (see below for why).

But this is a good question! And you already found the answer 🙂

> [@andyferris](#):
>
> Is it just to get the order of operations looking correct?

Yes, but just in case you haven’t gotten to the full explanation yet, it probably requires more words. A confusing (and interesting) part of transducers is that the flow of application is “flipped twice” (See also [Glossary](https://tkf.github.io/Transducers.jl/dev/#Glossary-1) in my docs or [the explanation in Clojure homepage](https://clojure.org/reference/transducers)).

A transducer is a function that maps a _reducing function_ to a reducing function. A _reducing function_ is the `op` argument you pass to normal `foldl` (e.g., `+` or `*`; a function of the form `(Y, X) -> Y` in general where `Y` is the “accumulator” and `X` is the “input”). Since transducer maps a function to a function, it is natural to define the composition by `∘` (and actually that’s how it works in Clojure). I could have used (and actually was using) `Filter(isodd) ∘ Map(double)` instead of `Filter(isodd) |> Map(double)`. Note that, as evident when `∘` is used, the output (= a function) of the _first_ transducer `Filter(isodd)` gets the input first. Maybe it becomes clear if you see how the application of this transducer to a reducing function (say) `+` works:

```julia
double(x) = 2x

# In hypothetical notation:
(Filter(isodd) |> Map(double))(+)(y, x)
== (Filter(isodd) ∘ Map(double))(+)(y, x) # `|>` actually means `∘` for transducers
== (Filter(isodd)(Map(double)(+)))(y, x) # expand `∘`
== if isodd(x) # expand `Filter`
     Map(double)(+)(y, x)
   else
     y
   end
== if isodd(x)
     (+)(y, double(x)) # expand `Map`
   else
     y
   end

```

As you can see, input `x` is `Filter`’ed first and then `Map`’ed.

I thought it would be “doubly” confusing and it could be easier to start using transducers if I use some “arrow like” symbol that points in the direction that the data flows.

Another reason is to make the API accessible in ASCII (although I totally like coding in Unicode personally).

But at the same time, I’m not entirely sure that this is the right interface, as it changes the semantics of the operator `|>` for this particular type. Is it something frowned upon in Julia API design? It seems there are not much other examples that “violate” operator semantics.

> [@andyferris](#):
>
> I’ve often wanted to have something like `∘` but writing the functions in the other order

FYI, I recently learned:

> Some authors compose in the opposite order, writing _fg_ or _f ∘ g_ for _g ∘ f_. Computer scientists using category theory very commonly write **_f ; g_ for _g ∘ f_** — [Category theory - Wikipedia](https://en.wikipedia.org/wiki/Category_theory#cite_note-6)

(although sadly we can’t use it in Julia)

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 14, 2019, 5:21am UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/14 "2019-01-14T05:21:07Z")

</div>

Transducers.jl is now registered 🎉. You can install it via `]add Transducers`.

I also added a [tutorial](https://tkf.github.io/Transducers.jl/dev/examples/tutorial_missings/) which shows how to use transducers while implementing various missing value handling.

---

<div class="post-metadata">

**Author:** ![ssfrr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ssfrr/32/3736_2.png) [@ssfrr](https://discourse.julialang.org/u/ssfrr)\
**Post date:** [January 16, 2019, 4:21pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/15 "2019-01-16T16:21:31Z")

</div>

The package documentation is really fantastic. Thanks for putting the work into it.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 17, 2019, 5:25am UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/16 "2019-01-17T05:25:27Z")

</div>

I should thank Documenter and Literate devs since those tools make writing docs super fun!

---

<div class="post-metadata">

**Author:** ![arghhhh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arghhhh/32/258_2.png) [@arghhhh](https://discourse.julialang.org/u/arghhhh)\
**Post date:** [January 17, 2019, 9:40pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/17 "2019-01-17T21:40:36Z")

</div>

I coded up something similar - iterator processing functions that didn’t have the source bound in. I used the Julia `nothing` value to represent out-of-input and no-more-output states. I think this is closer to the Julia iteration protocol. This seems simpler than the early termination and completing process mentioned in the Rich Hickey video.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 18, 2019, 1:54am UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/18 "2019-01-18T01:54:32Z")

</div>

Interesting. Is it something like a framework for higher-order functions composing a function that maps an iterator to an iterator?

If you use iterator protocol or something similar, I suppose that’s still “pull”-based framework (i.e., the user of the iterator processing functions is the one driving the loop)? If so, my guess is that it has the same pros and cons of the iterators. For example, can it have the performance equivalent to the manual loops for `flatten`/`Cat` (iterating over vector-of-vector), `zip`, `product`, and `BitArray`? With transducers, you can do so simply because you can write `foldl` which can have `for` loops using whatever the best looping strategy is. Of course, the compiler may be able to re-construct such loops from `Base.iterate`. But that sounds hard. (But I don’t know much about compilers.)

> [@arghhhh](#):
>
> I used the Julia `nothing` value to represent out-of-input and no-more-output states. I think this is closer to the Julia iteration protocol.

I don’t know how does it work in your iterator processing functions, but I think termination for both iterators and transducers are cumbersome enough that you’d write a macro for it anyway (there is [`IterTools.@ifsomething`](https://juliacollections.github.io/IterTools.jl/latest/#IterTools.@ifsomething-1)). If you use macros, I don’t think there is much difference. But I’d say writing `foldl` is much easier than writing `iterate`. See this comparison: [https://tkf.github.io/Transducers.jl/dev/examples/reducibles/](https://tkf.github.io/Transducers.jl/dev/examples/reducibles/)

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 18, 2019, 2:09am UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/19 "2019-01-18T02:09:02Z")

</div>

Another thought: I think the best “feature” of transducers for performance-oriented language like Julia is actually not transducers themselves but rather the _`foldl` implementations specialized for container types_. I guess that’s not news since Julia `Base` has very efficient `foldl`/`mapfoldl`. However, `foldl` (or `mapfoldl`) itself is not really composable — that’s where transducers come in. Transducers are composable “pre-processors” of the “reducing function” `op` you passed to `foldl`. I think one of the great observations by Rich Hickey is that this set of pre-processors can be as powerful as the usual iterator tool chain. But, as @arghhhh pointed out, it requires a generalization of `foldl` for supporting early termination and completion. I think it is worth doing so since it can be compiled away if you don’t use it.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 12, 2019, 5:50pm UTC](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159/20 "2019-03-12T17:50:26Z")

</div>

First, thanks for writing this package! I read through your code, watched the video, and read through your code again and it is starting to make sense, mostly because you organized and documented everything so nicely.

I have some specific questions:

1. My understanding is that the current implementation of `Base.collect` [here](https://github.com/tkf/Transducers.jl/blob/95520f0cb719827130eac7e912a7ec9d75e3e470/src/processes.jl#L469) uses type inference to infer the container type. Do you think it could use some strategy that has the same container type semantics as the default method, which widens on demand? Or does that not compose well with transducers?

2. Similarly, did you run into the type explosion problem [mentioned in the video](http://www.youtube.com/watch?v=6mTbuzafcII&t=28m6s) (_edit: corrected video link_) in the Julia implementation?

3. How should one go about implementing a custom container type that can accept output from transducer-mapped collections, what are the necessary primitives? Especially if a widening-on-demand strategy like `collect` is desired.

[Next page](https://discourse.julialang.org/t/ann-transducers-jl-efficient-and-composable-algorithms-for-map-and-reduce-like-operations/19159.md?page=2)