Multithreading and ProductIterator

Trying to use @threads to Base.Iterators.ProductIterator for optimization problem. Here is a simplified example:

x = 1:3;
y = 5:7;
Threads.@threads for i in Iterators.product(x, y)
    id = Threads.threadid();
    @info "$id <== $i"

It throws an exception:

ERROR: TaskFailedException:
MethodError: no method matching unsafe_getindex(::Base.Iterators.ProductIterator{Tuple{UnitRange{Int64},UnitRange{Int64}}}, ::Int64)

One simple workaround is using “collect”:

x = 1:3;
y = 5:7;
Threads.@threads for i in collect(Iterators.product(x, y))
    id = Threads.threadid();
    @info "$id <== $i"

Then it will work (with 6 cores in this example):

[ Info: 5 <== (2, 7)
[ Info: 4 <== (1, 7)
[ Info: 3 <== (2, 6)
[ Info: 2 <== (3, 5)
[ Info: 1 <== (1, 5)
[ Info: 6 <== (3, 7)
[ Info: 1 <== (2, 5)
[ Info: 3 <== (3, 6)
[ Info: 2 <== (1, 6)

Any better ways without collecting the list?

1 Like

I’d go something like

Threads.@threads for i in x
    for j in y
        id = Threads.threadid();
        @info "$id <== ($i, $j)"

This package could be helpful for parallel problems:

(FYI, Transducers.jl’s reduce does not support Iterators.product at the moment. It should be simple to add, though. Issue:


Maybe we can just use ((i,j) for i in x for j in y) - it seems to be Iterator.Flatten. Please note we can’t use int with @threads cause it has no length.

EDIT: now I see that Reducers.jl require it too, I cannot make it work.

I should have mentioned in my post that the real problem has more than two dimensions, [x1, x2, x3, x4, ... , x20]. So the Iterators.product will be a convenient scalable solution for this case. I am going to try Transducers.jl once it supports the Iterators.product.

You can also loop over single dimension with @threads and then loop over product of remaining dimensions. Spawning more threads than cores will not make your program faster.

It is good idea to loop over single dimension (of the longest length, for example) with @threads. Thank you.