# Multithreading and ProductIterator

**URL:** <https://discourse.julialang.org/t/multithreading-and-productiterator/33405>\
**Category:** General Usage\
**Tags:** multithreading\
**Created:** [January 15, 2020, 4:41pm UTC](https://discourse.julialang.org/t/multithreading-and-productiterator/33405 "2020-01-15T16:41:20Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yunde](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yunde/32/8874_2.png) [@Yunde](https://discourse.julialang.org/u/Yunde)\
**Post date:** [January 15, 2020, 4:41pm UTC](https://discourse.julialang.org/t/multithreading-and-productiterator/33405/1 "2020-01-15T16:41:20Z")

</div>

Trying to use @threads to Base.Iterators.ProductIterator for optimization problem. Here is a simplified example:

```julia
x = 1:3;
y = 5:7;
Threads.@threads for i in Iterators.product(x, y)
    id = Threads.threadid();
    @info "$id <== $i"
end

```

It throws an exception:

```
ERROR: TaskFailedException:
MethodError: no method matching unsafe_getindex(::Base.Iterators.ProductIterator{Tuple{UnitRange{Int64},UnitRange{Int64}}}, ::Int64)

```

One simple workaround is using “collect”:

```julia
x = 1:3;
y = 5:7;
Threads.@threads for i in collect(Iterators.product(x, y))
    id = Threads.threadid();
    @info "$id <== $i"
end

```

Then it will work (with 6 cores in this example):

```julia
[ Info: 5 <== (2, 7)
[ Info: 4 <== (1, 7)
[ Info: 3 <== (2, 6)
[ Info: 2 <== (3, 5)
[ Info: 1 <== (1, 5)
[ Info: 6 <== (3, 7)
[ Info: 1 <== (2, 5)
[ Info: 3 <== (3, 6)
[ Info: 2 <== (1, 6)

```

Any better ways without collecting the list?

---

<div class="post-metadata">

**Author:** ![Jakub\_Wronowski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakub_wronowski/32/204030_2.png) [@Jakub\_Wronowski](https://discourse.julialang.org/u/Jakub_Wronowski)\
**Post date:** [January 15, 2020, 8:07pm UTC](https://discourse.julialang.org/t/multithreading-and-productiterator/33405/2 "2020-01-15T20:07:08Z")

</div>

I’d go something like

```julia
Threads.@threads for i in x
    for j in y
        id = Threads.threadid();
        @info "$id <== ($i, $j)"
    end
end

```

This package could be helpful for parallel problems:

> [@Thread- and process-based parallelisms in Transducers.jl (+ some news)](https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285):
>
> It’s been a while since I added parallelism supports in Transducers.jl but I’ve never announced this feature properly. I just added a few utility functions and a tutorial so I think it’s good timing to do this. Quoting [Overview of parallel processing in Transducers.jl](https://tkf.github.io/Transducers.jl/dev/parallelism/): Transducers.jl supports thread-based ([reduce](https://tkf.github.io/Transducers.jl/dev/manual/#Base.reduce)) and process-based ([dreduce](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.dreduce)) parallelisms with the same composable API; i.e. transducers. Having a uniform API to cover different parallelisms as well as sequential processing [fold…](https://tkf.github.io/Transducers.jl/dev/manual/#Base.foldl)

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [January 15, 2020, 8:33pm UTC](https://discourse.julialang.org/t/multithreading-and-productiterator/33405/3 "2020-01-15T20:33:08Z")

</div>

(FYI, Transducers.jl’s `reduce` does not support `Iterators.product` at the moment. It should be simple to add, though. Issue: [https://github.com/tkf/Transducers.jl/issues/152](https://github.com/tkf/Transducers.jl/issues/152))

---

<div class="post-metadata">

**Author:** ![Jakub\_Wronowski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakub_wronowski/32/204030_2.png) [@Jakub\_Wronowski](https://discourse.julialang.org/u/Jakub_Wronowski)\
**Post date:** [January 15, 2020, 8:37pm UTC](https://discourse.julialang.org/t/multithreading-and-productiterator/33405/4 "2020-01-15T20:37:40Z")

</div>

Maybe we can just use `((i,j) for i in x for j in y)` - it seems to be Iterator.Flatten. Please note we can’t use int with @threads cause it has no `length`.

EDIT: now I see that Reducers.jl require it too, I cannot make it work.

---

<div class="post-metadata">

**Author:** ![Yunde](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yunde/32/8874_2.png) [@Yunde](https://discourse.julialang.org/u/Yunde)\
**Post date:** [January 16, 2020, 4:03am UTC](https://discourse.julialang.org/t/multithreading-and-productiterator/33405/5 "2020-01-16T04:03:31Z")

</div>

I should have mentioned in my post that the real problem has more than two dimensions, `[x1, x2, x3, x4, ... , x20]`. So the `Iterators.product` will be a convenient scalable solution for this case. I am going to try `Transducers.jl` once it supports the `Iterators.product`.

---

<div class="post-metadata">

**Author:** ![Jakub\_Wronowski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakub_wronowski/32/204030_2.png) [@Jakub\_Wronowski](https://discourse.julialang.org/u/Jakub_Wronowski)\
**Post date:** [January 16, 2020, 7:10am UTC](https://discourse.julialang.org/t/multithreading-and-productiterator/33405/6 "2020-01-16T07:10:48Z")

</div>

You can also loop over single dimension with @threads and then loop over product of remaining dimensions. Spawning more threads than cores will not make your program faster.

---

<div class="post-metadata">

**Author:** ![Yunde](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yunde/32/8874_2.png) [@Yunde](https://discourse.julialang.org/u/Yunde)\
**Post date:** [January 16, 2020, 12:30pm UTC](https://discourse.julialang.org/t/multithreading-and-productiterator/33405/7 "2020-01-16T12:30:29Z")

</div>

It is good idea to loop over single dimension (of the longest length, for example) with `@threads`. Thank you.