Multithreaded compute farm

If ThreadPools.jl fits your need, I think it’s great and it’s not my intention to discourage it at all. All I want to do is to properly characterize what it actually is (or was, when I checked the last time) for helping you and other readers to understand the use case of it. I also didn’t say it’s not possible to use or implement “thread pool.” I was trying to explain that you don’t need ThreadPools.jl for this and ThreadPools.jl introduces overheads since it tries to disable what Julia runtime does (e.g., task migration across OS threads, in Julia 1.7 and later). There are much simpler and composable ways to implement task pools. In particular, the code in the OP already does it.

2 Likes