I have a C code (a monte carlo simulation) with lots of global state. I would like to call multiple independent copies of this code from julia (so it can be safely parallelized). Is this possible?
Julia does not allow you to do it more than what you could within C.
Some platforms has some form of support for this in dlopen flags so you can have a look at that. IIRC the support is usually limited (you may not be open an arbitrary number of copies) and the overhead is definitely going to be very big so it’s much better to fix the c library.
Actually it was not
dlopen flag, but
dlmopen on linux. I did remembered correctly about the limitation
The glibc implementation supports a maximum of 16 namespaces.
Well, you could start multiple processes and call the library independently from each process.
That might be an option. So you are refering to things described here:
One thing I need to do is passing closures to the C code. I am not so experienced with multiprocessing in julia. But AFAICT this needs a lot of toplevel
So I could not hide this in a library with an api like
using MyCCode simulation(nproc=10) do state # interact with simulation end
But it would look like
@everywhere using MyCCode @everywhere function mycallback(state) ... end combine_results(map(1:10) do iproc remotecall(single_simulation, iproc, mycallback) end
I think the easiest way is to put everything in a package and the load it using
@everywhere using ...., then
remotecall that. My experience with multi-processing is that it requires a bit of experimentation to get things working.