# Multithreaded mapping of an iterator (not a collection)

**URL:** <https://discourse.julialang.org/t/multithreaded-mapping-of-an-iterator-not-a-collection/47906>\
**Category:** New to Julia\
**Created:** [October 7, 2020, 1:04am UTC](https://discourse.julialang.org/t/multithreaded-mapping-of-an-iterator-not-a-collection/47906 "2020-10-07T01:04:47Z")\
**Posts on this page:** 1\
**Showing post:** 9

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [October 7, 2020, 9:45pm UTC](https://discourse.julialang.org/t/multithreaded-mapping-of-an-iterator-not-a-collection/47906/9 "2020-10-07T21:45:41Z")

</div>

> [@yarnton](#):
>
> Using `ThreadPools.tmap` , which is a drop-in replacement for `map` .

If `ThreadPools.tmap` fits your use-case, then I think that’s great! But I’d point out a few caveats:

- It looks like `ThreadPools.tmap` [collects input into an array](https://github.com/tro3/ThreadPools.jl/blob/bd329839c2c0968356029223ab2a0c53ed6de1bb/src/staticpool.jl#L34) first. It’s a robust strategy but not very optimal. For example, you cannot interleave the iteration and the computation this way.
- It [pre-computes output element type](https://github.com/tro3/ThreadPools.jl/blob/bd329839c2c0968356029223ab2a0c53ed6de1bb/src/staticpool.jl#L37) using [the compiler internal](https://github.com/tro3/ThreadPools.jl/blob/bd329839c2c0968356029223ab2a0c53ed6de1bb/src/interface.jl#L11). Thus, it is not a [typocalypse](https://discourse.julialang.org/search?q=typocalypse)-free solution. Practically, it means that updating `julia` _can_ break your code.
- The main goal of ThreadPools.jl is to separate latency-critical code from throughput-oriented code. ThreadPools.jl achieves this by a clever trick but unfortunately this impedes dynamic scheduling by the `julia` runtime. As a result, using this at library level means we will loose composable nested parallelism ecosystem that Julia’s parallel runtime is designed to support.

> [@yarnton](#):
>
> I still don’t get why the example @stillyslalom works with `ThreadsX.map` but it doesn’t on a `FASTA.reader` .

`ThreadsX.map` converts the iterator transformations (e.g., `Iterators.filter`) to [transducers](https://github.com/JuliaFolds/Transducers.jl) and runs the reduction on the “inner most” iterator. So, if you have `ThreadsX.map(f, (x for x in xs if p(x)))` (or equivalently `ThreadsX.map(f, Iterators.filter(p, xs))`), what matters is if the iterator (collection) `xs` supports SplittablesBase.jl API.

---

_[View the full topic](https://discourse.julialang.org/t/multithreaded-mapping-of-an-iterator-not-a-collection/47906)._