# Run multiple python instances with pycall in different threads

**URL:** <https://discourse.julialang.org/t/run-multiple-python-instances-with-pycall-in-different-threads/78399>\
**Category:** General Usage\
**Tags:** pycall, parallel\
**Created:** [March 24, 2022, 2:38pm UTC](https://discourse.julialang.org/t/run-multiple-python-instances-with-pycall-in-different-threads/78399 "2022-03-24T14:38:41Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tyde](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tyde/32/922_2.png) [@Tyde](https://discourse.julialang.org/u/Tyde)\
**Post date:** [March 24, 2022, 2:38pm UTC](https://discourse.julialang.org/t/run-multiple-python-instances-with-pycall-in-different-threads/78399/1 "2022-03-24T14:38:41Z")

</div>

Hi,

I have a project where I need to call python code via PyCall.jl. This python code itself does interact with a third-party library (In this case I use pyspice the control the simulation of multiple circuits). To increase the speed of my script I would like to run multiple threads that each control one instance of python so that the python processes also run in parallel.

I tested just running the PyCall in parallel via pmap or other Distributed.jl directives, but it seems that it would all run in one python instance. Is there an option to “spawn” a python instance?

Thanks,  
Daniel

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [March 24, 2022, 3:41pm UTC](https://discourse.julialang.org/t/run-multiple-python-instances-with-pycall-in-different-threads/78399/2 "2022-03-24T15:41:40Z")

</div>

I don’t have a working PyCall to hand, but with PythonCall the following works fine, and I’d expect PyCall to work just the same:

```julia-repl
julia> @everywhere using PythonCall

julia> @time pmap(1:4) do i
           __main__ , time = pyimport(" __main__", "time")
           time.sleep(4 - i)
           __main__.foo = myid()
           time.sleep(i)
           i => __main__.foo
       end
  4.284934 seconds (268.97 k allocations: 14.611 MiB, 3.89% compilation time)
4-element Vector{Pair{Int64, Py}}:
 1 => 5
 2 => 2
 3 => 3
 4 => 4

```

In this code, each iteration sleeps for a total of 4 seconds, but sets ` __main__.foo` to the worker ID at a different time, then returns ` __main__.foo` at the end. The total run-time is just over 4 seconds so they are indeed in parallel. Since they return different numbers, they are running separate interpreters (if they were running the same interpreter, ` __main__.foo` would be the same number in all workers.)

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 24, 2022, 3:50pm UTC](https://discourse.julialang.org/t/run-multiple-python-instances-with-pycall-in-different-threads/78399/3 "2022-03-24T15:50:37Z")

</div>

> [@cjdoris](#):
>
> `pmap(1:4) do i`

Note those are different _processes_ (distributed memory), which is totally distinct from different shared-memory “threads” — it’s really crucial to understand the difference if you are doing parallel programming!

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [March 24, 2022, 3:53pm UTC](https://discourse.julialang.org/t/run-multiple-python-instances-with-pycall-in-different-threads/78399/4 "2022-03-24T15:53:56Z")

</div>

The OP specifically mentioned `pmap` and `Distributed` so I answered to that (multi-processing).

If you want to use multi-_threading_ then the short answer is no, you can’t use PyCall (or PythonCall) with that. The long answer is yes, maybe, if you’re very very careful with the GIL.

---

<div class="post-metadata">

**Author:** ![martinmestre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/martinmestre/32/215857_2.png) [@martinmestre](https://discourse.julialang.org/u/martinmestre)\
**Post date:** [August 16, 2023, 8:22pm UTC](https://discourse.julialang.org/t/run-multiple-python-instances-with-pycall-in-different-threads/78399/5 "2023-08-16T20:22:08Z")

</div>

Hi, could you please tell me what should be done with the GIL for multithreading?

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [August 16, 2023, 8:43pm UTC](https://discourse.julialang.org/t/run-multiple-python-instances-with-pycall-in-different-threads/78399/6 "2023-08-16T20:43:22Z")

</div>

> [@martinmestre](#):
>
> could you please tell me what should be done with the GIL for multithreading?

See [Dropping GIL when calling Julia from Python (JuliaCall) · Issue #343 · JuliaPy/PythonCall.jl · GitHub](https://github.com/cjdoris/PythonCall.jl/issues/343)

But general if you want to parallelize Python calls it is best to do it from multiple processes (as in the examples above), not multiple threads (for which little or no Python parallelism is possible).

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [August 16, 2023, 9:07pm UTC](https://discourse.julialang.org/t/run-multiple-python-instances-with-pycall-in-different-threads/78399/7 "2023-08-16T21:07:45Z")

</div>

2 posts were split to a new topic: [Distributed-memory parallelism on a shared-memory machine](https://discourse.julialang.org/t/distributed-memory-parallelism-on-a-shared-memory-machine/102883)

---

<div class="post-metadata">

**Author:** ![hhaensel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hhaensel/32/1207_2.png) [@hhaensel](https://discourse.julialang.org/u/hhaensel)\
**Post date:** [June 13, 2024, 7:06am UTC](https://discourse.julialang.org/t/run-multiple-python-instances-with-pycall-in-different-threads/78399/8 "2024-06-13T07:06:11Z")

</div>

It seems that Python can’t be made running in different threads, but there’s a workaround to pin python routines to a fixed thread just in case your application (e.g. a multi-user web application like GenieFramwork), will use multi-threading to handle different user requests.  
I posted a solution in a [related thread](https://discourse.julialang.org/t/julia-crashed-when-using-pycall-in-multiple-threads/111004/9)