How to get around lack of indexing to Base.Iterators.product to allow threaded access

New to Julia
,
1

The following loop works without threading but fails because it cant seem to index the iterator:

arr1 = [5:10;]
arr2 = [20:5:30;]
comb = Base.Iterators.product(arr1,arr2)

function threadedComb(comb)
    Threads.@threads for c in comb
        println(c)
    end
end

function singleComb(comb)
    for c in comb
        println(c)
    end
end
Without threading 
singleComb(comb)
(5, 20)
(6, 20)
(7, 20)
(8, 20)
(9, 20)
(10, 20)
(5, 25)
(6, 25)
(7, 25)
(8, 25)
(9, 25)
(10, 25)
(5, 30)
(6, 30)
(7, 30)
(8, 30)
(9, 30)
(10, 30)

With threading:

threadedComb(comb)
ERROR: TaskFailedException:
MethodError: no method matching firstindex(::Base.Iterators.ProductIterator{Tuple{Array{Int64,1},Array{Int64,1}}})
Closest candidates are:
  firstindex(::Core.SimpleVector) at essentials.jl:601
  firstindex(::Cmd) at process.jl:639
  firstindex(::Base64.Buffer) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Base64/src/buffer.jl:18
  ...
Stacktrace:
 [1] (::var"#409#threadsfor_fun#37"{Base.Iterators.ProductIterator{Tuple{Array{Int64,1},Array{Int64,1}}}})(::Bool) at ./threadingconstructs.jl:46
 [2] (::var"#409#threadsfor_fun#37"{Base.Iterators.ProductIterator{Tuple{Array{Int64,1},Array{Int64,1}}}})() at ./threadingconstructs.jl:28
Stacktrace:
 [1] wait(::Task) at ./task.jl:267
 [2] macro expansion at ./threadingconstructs.jl:69 [inlined]
 [3] threadedComb(::Base.Iterators.ProductIterator{Tuple{Array{Int64,1},Array{Int64,1}}}) at /mnt/evo512/insync/Software_a1/testCUDA/test_threaded_combos.jl:8
 [4] top-level scope at /mnt/evo512/insync/Software_a1/testCUDA/test_threaded_combos.jl:48

How can I modify threadedComb to allow threaded access?

2 
function threadedComb(comb)
    Threads.@threads for c in Tuple(comb)
        println(c)
    end
end

works

3

you need to make it “not iterator” first because threads would access later index at the same time; a common function to use is collect

1 Like
4

Thanks @jling, is this implicitly done in the solution by @pbayer? I like the tuple method because of immutability.

5

yes:

...
In _totuple(#unused#, itr, s) at tuple.jl:258
>258  _totuple(::Type{Tuple}, itr, s...) = (collect(Iterators.rest(itr,s...))...,)

julia> @btime Tuple($comb);
  596.106 ns (20 allocations: 1.22 KiB)

julia> @btime collect($comb);
  39.652 ns (1 allocation: 368 bytes)

and because Tuple is collet then splat (in this case), it’s slower

1 Like
6

FYI GitHub - JuliaFolds/FLoops.jl: Fast sequential, threaded, and distributed for-loops for Julia—fold for humans™ and https://github.com/tkf/ThreadsX.jl support Iterators.product. They can also be used with other iterable types supported by https://github.com/JuliaFolds/SplittablesBase.jl. It may be better than collecting the iterator first when the resulting array is large.

2 Likes