I have a C code (a monte carlo simulation) with lots of global state. I would like to call multiple independent copies of this code from julia (so it can be safely parallelized). Is this possible?
Julia does not allow you to do it more than what you could within C.
Some platforms has some form of support for this in dlopen flags so you can have a look at that. IIRC the support is usually limited (you may not be open an arbitrary number of copies) and the overhead is definitely going to be very big so it’s much better to fix the c library.
Actually it was not dlopen
flag, but dlmopen
on linux. I did remembered correctly about the limitation
The glibc implementation supports a maximum of 16 namespaces.
Well, you could start multiple processes and call the library independently from each process.
That might be an option. So you are refering to things described here:
https://docs.julialang.org/en/v1/manual/parallel-computing/#Multi-Core-or-Distributed-Processing-1
One thing I need to do is passing closures to the C code. I am not so experienced with multiprocessing in julia. But AFAICT this needs a lot of toplevel @everywhere
code?
So I could not hide this in a library with an api like
using MyCCode
simulation(nproc=10) do state
# interact with simulation
end
But it would look like
@everywhere using MyCCode
@everywhere function mycallback(state)
...
end
combine_results(map(1:10) do iproc
remotecall(single_simulation, iproc, mycallback)
end
?
I think the easiest way is to put everything in a package and the load it using @everywhere using ....
, then remotecall
that. My experience with multi-processing is that it requires a bit of experimentation to get things working.