# Materialize transducer into existing storage

**URL:** <https://discourse.julialang.org/t/materialize-transducer-into-existing-storage/87258>\
**Category:** General Usage\
**Tags:** transducers\
**Created:** [September 14, 2022, 7:46am UTC](https://discourse.julialang.org/t/materialize-transducer-into-existing-storage/87258 "2022-09-14T07:46:09Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mfiano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mfiano/32/37565_2.png) [@mfiano](https://discourse.julialang.org/u/mfiano)\
**Post date:** [September 14, 2022, 7:46am UTC](https://discourse.julialang.org/t/materialize-transducer-into-existing-storage/87258/1 "2022-09-14T07:46:09Z")

</div>

I defined a [reducible](https://juliafolds.github.io/Transducers.jl/stable/reference/interface/#Interface-for-reducibles) for my `Kernel` type, which works fine, and I can materialize pipelines with `collect`, etc.

I now want to be able to write a function that does `kernel |> Map(func) |> # ... materialize into existing vector storage`.

Basically I want to do what `Base.map!` does to eagerly construct a collection from a transducer pipeline, but storing the result in a destination collection without an intermediary allocation. I see there is a `map!` in Transducers.jl, but it seems to want 3 arguments, and being new to Transducers.jl, I really have no idea how to work with that, or if it’s even what I’m looking for.

Thanks for any help!

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [September 14, 2022, 9:24am UTC](https://discourse.julialang.org/t/materialize-transducer-into-existing-storage/87258/2 "2022-09-14T09:24:05Z")

</div>

Is your `Kernel <: AbstractArray`? If so, `map!(Map(func), dest, kernel)` should work fine, no? Otherwise, `sizehint!`ing an empty `dest` array and using `append!(Map(func), dest, kernel)` or `append!!(dest, kernel |> Map(func))` should work, I think.

---

<div class="post-metadata">

**Author:** ![mfiano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mfiano/32/37565_2.png) [@mfiano](https://discourse.julialang.org/u/mfiano)\
**Post date:** [September 14, 2022, 1:43pm UTC](https://discourse.julialang.org/t/materialize-transducer-into-existing-storage/87258/4 "2022-09-14T13:43:36Z")

</div>

```julia
julia> map!(Map(identity), dest, kernel)

```

The above hangs my REPL forever. No stack overflow, just no return value with heavy CPU usage for something that normally takes nanoseconds.

Edit: Aborting the evaluation I get:

```julia
ERROR: InterruptException:
Stacktrace:
 [1] map!(f::Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{M
ap{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map
{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{Map{typeof(identity)}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}, dest::Vector{Cell}, kernel::Kernel
{Rect}) (repeats 890 times)
   @ DungeonGenerator ~/projects/Julia/DungeonGenerator/src/DungeonGenerator.jl:80
 [2] top-level scope
   @ REPL[15]:1

```

The latter does work, and I was able to do this on my own previously, but obviously, instead of re-using existing storage, it grows the vector. Sure I could `sizehint!` it to prevent allocation, but the goal is to just do a memory copy and retain the length of the destination vector, or at least start writing from the beginning rather than the end.

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [September 14, 2022, 2:16pm UTC](https://discourse.julialang.org/t/materialize-transducer-into-existing-storage/87258/5 "2022-09-14T14:16:09Z")

</div>

Now that I read that again: Shouldn’t it just be `map!(func, dest, kernel)`?

Side note: I have never used Transducers before, so it’s unlikely anything useful is going to come out of me blindly suggesting things without access to the original codebase 😃

---

<div class="post-metadata">

**Author:** ![mfiano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mfiano/32/37565_2.png) [@mfiano](https://discourse.julialang.org/u/mfiano)\
**Post date:** [September 14, 2022, 2:38pm UTC](https://discourse.julialang.org/t/materialize-transducer-into-existing-storage/87258/6 "2022-09-14T14:38:00Z")

</div>

No that won’t do, as a transducer is not a function. I finally figured something out though that is type-stable, doesn’t allocate, and is extremely fast (thanks to @jakobnissen for the starting point):

```julia
function Base.map!(f::F, dest, kernel::Kernel) where {F}
    foldl(kernel |> Map(f) |> Enumerate(); init=dest) do v, (i, e)
        i > length(dest) && resize!(dest, i)
        v[i] = e
        v
    end
end

```

If anyone sees any room for improvement, I’d like to know, being relatively new to Julia, and extremely new to Transducers.jl myself. Thank you!