Hi, is it possible to deactivate (process) parallelism for a specific function call, i.e. force running this specific portion of the code on a single process even though it could potentially be distributed? My problem is that I have an outer @parallel for loop, and that I have inside the loop a call to a module which is by default parallelized. In particular this module detects that I am running Julia with more than 1 process and automatically tries to use multiple processes… but it doesn’t work (because all processes are busy I guess… I get an error message “process x didn’t connect to y within 60 seconds”).
So my question is: is there an automated way to avoid that, or should each package developer manually provide an option to activate/deactivate parallelism?
Can you point out which package you’re using, and how you’re calling it? It is possible this is just a bug.
To answer your question, there isn’t really an easy way to do this right now (and we have a somewhat similar issue with Threads.@threads; in that case, multithreading is internally disabled in the inner loop, which isn’t necessarily optimal). I think you have three options:
- Modify the library in question to allow disabling parallelism (difficulty: probably easy).
- Modify Distributed to have some sort of “no parallelism region” (difficulty: medium/hard, and not so likely to get accepted).
- Convince the library maintainer to use a higher-level abstraction for distributed computing, like Dagger.jl, which handles parallelism for you, and which is arguably the more composable and scalable approach (note: I am the Dagger maintainer).
I personally advocate for approach 3. If the library in question used Dagger for its computation, then Dagger’s built-in scheduler would handle distributing work for you. When you then wish to include the library’s distributed code within your own distributed code, you’d simply need them to expose their “Dagger graph” to you, and you could plug it into your own Dagger graph, and have everything “just work”. I’d be happy to help with this if you can show me your code, and point out which library you’re calling and how you’re calling it.
Thanks for your answer, I wasn’t aware of Dagger.jl, it’s interesting I’ll have a closer look at it. In my case I cannot provide a simple MWE but I essentially have a @sync @distributed for loop inside which I use the GaussianMixture package (with many non-distributed calls in between, but I guess these are not relevant). This package relies on e.g. pmap and a few other parallel constructions. I guess as you say the easiest in a first time would be to go for option 1 (I understand the interest of Dagger, but naturally it takes a bit of time to adapt everything, and in my case this is really not central).
Regarding the fact that multithreading is disabled in inner loops, this is a design choice to do it only for threads and not for processes? Or it should be the same and there is a bug somewhere?