Calling Julia from Python on thread

I’ve been playing around with calling Julia functions from Python which has all worked fine until I try and call from a thread. To illustrate, here is a short example:

Julia code:

function add(x, y)
    return x + y
end

Python code:

import threading
import time
import julia
from julia import Main     # Move these two lines to add_numbers_from_julia() to call from thread
Main.include('library.jl') # ..but it will only work *once* :|

def add_numbers_from_julia():
    z = Main.add(4, 5)
    print(f"result = {z}")

def main():
    i = -1
    while i != 0:
        print("Options:")
        print("0 - Exit")
        print("1 - Add 4 and 5")
        print("2 - Add 4 and 5 from thread")

        i = int(input())

        if i == 1:
            add_numbers_from_julia()
        elif i == 2:
            thread = threading.Thread(target = add_numbers_from_julia, args = ())
            thread.start()
            time.sleep(5)
        print()
            

if __name__ == "__main__":
    main()

In the above code, if the user chooses option 1 , we call the Julia add(x,y) function on the current thread and it all works fine.

If the user chooses option 2 we attempt to call add(x,y) from a thread, which causes an access violation. If we remove the from julia import Main and Main.include('library.jl') lines from the top of the file and put them into the add_numbers_from_julia() function, then we can successfully call add(x,y) but only once . Attempting to hit 2 again will cause another access violation.

Can someone tell me whats going wrong and where I should be importing the Julia code if I want to call it from a thread?

Many TIA

1 Like

I encountered a similar issue some time ago. PyJulia is not thread-safe, see here:

https://github.com/dask/dask/issues/5732

However, it should work if you use Multiprocessing on Python-side.

However, it should work if you use Multiprocessing on Python-side.

Need to be careful here because the default multiprocessing mode forks the process, which tends to break things in a multithreaded program which does not expect a fork. Use multiprocessing.set_start_method("spawn") to launch independent processes, which should safe.