Ccall into independent copies C library


#1

I have a C code (a monte carlo simulation) with lots of global state. I would like to call multiple independent copies of this code from julia (so it can be safely parallelized). Is this possible?


#2

Julia does not allow you to do it more than what you could within C.

Some platforms has some form of support for this in dlopen flags so you can have a look at that. IIRC the support is usually limited (you may not be open an arbitrary number of copies) and the overhead is definitely going to be very big so it’s much better to fix the c library.


#3

Actually it was not dlopen flag, but dlmopen on linux. I did remembered correctly about the limitation

The glibc implementation supports a maximum of 16 namespaces.


#4

Well, you could start multiple processes and call the library independently from each process.


#5

That might be an option. So you are refering to things described here:
https://docs.julialang.org/en/v1/manual/parallel-computing/#Multi-Core-or-Distributed-Processing-1

One thing I need to do is passing closures to the C code. I am not so experienced with multiprocessing in julia. But AFAICT this needs a lot of toplevel @everywhere code?
So I could not hide this in a library with an api like

using MyCCode
simulation(nproc=10) do state
       # interact with simulation
end

But it would look like

@everywhere using MyCCode
@everywhere function mycallback(state)
       ...
end

combine_results(map(1:10) do iproc
    remotecall(single_simulation, iproc, mycallback)
end

?


#6

I think the easiest way is to put everything in a package and the load it using @everywhere using ...., then remotecall that. My experience with multi-processing is that it requires a bit of experimentation to get things working.