Multithreading and ProductIterator

General Usage
1

Trying to use @threads to Base.Iterators.ProductIterator for optimization problem. Here is a simplified example:

x = 1:3;
y = 5:7;
Threads.@threads for i in Iterators.product(x, y)
    id = Threads.threadid();
    @info "$id <== $i"
end

It throws an exception:

ERROR: TaskFailedException:
MethodError: no method matching unsafe_getindex(::Base.Iterators.ProductIterator{Tuple{UnitRange{Int64},UnitRange{Int64}}}, ::Int64)

One simple workaround is using “collect”:

x = 1:3;
y = 5:7;
Threads.@threads for i in collect(Iterators.product(x, y))
    id = Threads.threadid();
    @info "$id <== $i"
end

Then it will work (with 6 cores in this example):

[ Info: 5 <== (2, 7)
[ Info: 4 <== (1, 7)
[ Info: 3 <== (2, 6)
[ Info: 2 <== (3, 5)
[ Info: 1 <== (1, 5)
[ Info: 6 <== (3, 7)
[ Info: 1 <== (2, 5)
[ Info: 3 <== (3, 6)
[ Info: 2 <== (1, 6)

Any better ways without collecting the list?

1 Like
2

I’d go something like

Threads.@threads for i in x
    for j in y
        id = Threads.threadid();
        @info "$id <== ($i, $j)"
    end
end

This package could be helpful for parallel problems:

3

(FYI, Transducers.jl’s reduce does not support Iterators.product at the moment. It should be simple to add, though. Issue: https://github.com/tkf/Transducers.jl/issues/152)

3 Likes
4

Maybe we can just use ((i,j) for i in x for j in y) - it seems to be Iterator.Flatten. Please note we can’t use int with @threads cause it has no length.

EDIT: now I see that Reducers.jl require it too, I cannot make it work.

5

I should have mentioned in my post that the real problem has more than two dimensions, [x1, x2, x3, x4, ... , x20]. So the Iterators.product will be a convenient scalable solution for this case. I am going to try Transducers.jl once it supports the Iterators.product.

6

You can also loop over single dimension with @threads and then loop over product of remaining dimensions. Spawning more threads than cores will not make your program faster.

7

It is good idea to loop over single dimension (of the longest length, for example) with @threads. Thank you.