I have a project where I need to call python code via PyCall.jl. This python code itself does interact with a third-party library (In this case I use pyspice the control the simulation of multiple circuits). To increase the speed of my script I would like to run multiple threads that each control one instance of python so that the python processes also run in parallel.
I tested just running the PyCall in parallel via pmap or other Distributed.jl directives, but it seems that it would all run in one python instance. Is there an option to “spawn” a python instance?
I don’t have a working PyCall to hand, but with PythonCall the following works fine, and I’d expect PyCall to work just the same:
julia> @everywhere using PythonCall
julia> @time pmap(1:4) do i
__main__, time = pyimport("__main__", "time")
time.sleep(4 - i)
__main__.foo = myid()
time.sleep(i)
i => __main__.foo
end
4.284934 seconds (268.97 k allocations: 14.611 MiB, 3.89% compilation time)
4-element Vector{Pair{Int64, Py}}:
1 => 5
2 => 2
3 => 3
4 => 4
In this code, each iteration sleeps for a total of 4 seconds, but sets __main__.foo to the worker ID at a different time, then returns __main__.foo at the end. The total run-time is just over 4 seconds so they are indeed in parallel. Since they return different numbers, they are running separate interpreters (if they were running the same interpreter, __main__.foo would be the same number in all workers.)
Note those are different processes (distributed memory), which is totally distinct from different shared-memory “threads” — it’s really crucial to understand the difference if you are doing parallel programming!
The OP specifically mentioned pmap and Distributed so I answered to that (multi-processing).
If you want to use multi-threading then the short answer is no, you can’t use PyCall (or PythonCall) with that. The long answer is yes, maybe, if you’re very very careful with the GIL.
But general if you want to parallelize Python calls it is best to do it from multiple processes (as in the examples above), not multiple threads (for which little or no Python parallelism is possible).
It seems that Python can’t be made running in different threads, but there’s a workaround to pin python routines to a fixed thread just in case your application (e.g. a multi-user web application like GenieFramwork), will use multi-threading to handle different user requests.
I posted a solution in a related thread