How to parallelize list comprehension and map with multithreading?

General Usage
10

Apologies, I should have given a complete example. The issue you’re coming on is that the elements of Iterators.product(...) are tuples. You could splat them, with something like

ThreadsX.map(z->my_function(z...), Iterators.product(Xs, Ys))

or you could make a new method for my_function and then call as you were before:

my_function(xy) = my_function(xy[1], xy[2])
ThreadsX.map(my_function, Iterators.product(Xs, Ys))

I haven’t benchmarked these things and so maybe using 2D generators is better for performance, but I would guess the performance should be similar. ThreadsX/the whole transducer ecosystem seems (very impressively) capable of handling pretty complicated iterators and still making good use of however many threads you give it.

2 Likes