How to parallelize list comprehension and map with multithreading?

If your end goal is to use functions invoking HDF5 API, you need to figure out if its API allows concurrent access. There is no other way to make it work with threading. Otherwise, use DistributedEx() by passing it to the last argument of Folds API.

If you are trying to understand when and how to use threading, see the docstring examples in Folds.jl · Folds. These docstrings “typically” contain two examples. The first one is for demonstrating the functionality with a small size problem. So, there is no benefits in using threading (e.g., sum(1:10000)). The second one is complicated enough to see some speedups with multiple threads (at least in my machine when I tried). These example may give you some ideas when/how to use threading. Typically, you need a serial computation that takes at least about 10 μs.

(But your last example with ThreadsX.map and Folds.map should produce some speedups. It’s simply a performance bug that I need to fix.)

Also, it’s better idea to share the value of Threads.nthreads() (and nprocs() if using Distributed) for sharing the parallel benchmark result.

3 Likes