Not able to run Julia through Flask-Celery

[Palli]

We integrated julia using a python library called julia==0.6.0

It seems you’re using PyJulia package, but note its docs, under “Limitations” (your error is likely a race condition because of the given limitation of the software you use):

No threading support

PyJulia cannot be used in different threads since libjulia is not thread safe. However, you can use multiple threads within Julia. For example, start IPython by JULIA_NUM_THREADS=4 ipython and then run:

In [1]: %load_ext julia.magic Initializing Julia runtime. This may take some time… In [2]: %%julia …: a = zeros(10) …: Threads.@threads for i = 1:10 …: a[i] = Threads.threadid() …: end …: a Out[3]: array([1., 1., 1., 2., 2., 2., 3., 3., 4., 4.])

PyJulia does not release GIL

PyJulia does not release the Global Interpreter Lock (GIL) while calling Julia functions since PyCall expects the GIL to be acquired always. It means that Python code and Julia code cannot run in parallel.

I recognize the error you have, as something coming from Julia without the “celery_worker_1 |” prefix, which I believe the Python side added somehow. I suppose that’s celery worker 1 (of many), i.e. a thread?

You can use all Python libraries from Julia (i.e. Julia is your main language) through PythonCall.jl (or older PyCall.jl). With it or PyJulia, you can also use all Julia libraries from Python. In both cases some limitations may apply:

When you e.g. call Julia from Python, the library calling may also be a framework. Frameworks call you (“the Hollywood principle” of frameworks), so they must run in the “top” language, as you are doing with Flask.

You may want to consider some popular Julia (web) framework such as Genie.jl as a replacement for Python’s Flask (and replace all Python framework stuff), or e.g. this alternative:

Dance.jl - Python Flask equivalent web framework

Hello Julia community, Am pleased to announce a new web framework focused on facilitating process of outputting Julia computed data to the web very easily. Feel that reason Python is still more mainstream than Julia is simply due to complete ecosystem of tools. More precisely many data scientists can easily use Flask to share their data with outside world. Not to that mention staying with 1 backend language is simply easier for many large teams. Which is why I have developed Dance.jl along wit…

Because serving web pages means a lot of requests come in and you want to serve them as fast as possible, meaning in parallel (“asynchronous”), often done with threading, something those interop libraries may not support. Unless possible if you jump through some hoops. Something you can avoid if using a full Julia (or full Python) solution.

Dance.jl - Python Flask equivalent web framework

Hello Julia community, Am pleased to announce a new web framework focused on facilitating process of outputting Julia computed data to the web very easily. Feel that reason Python is still more mainstream than Julia is simply due to complete ecosystem of tools. More precisely many data scientists can easily use Flask to share their data with outside world. Not to that mention staying with 1 backend language is simply easier for many large teams. Which is why I have developed Dance.jl along wit…

Compatibility Tools · PythonCall & JuliaCall

Asynchronous Julia code (including Makie)

Asynchronous Julia code will not normally run while Python is executing, unless it is in a separate thread.

This can be fixed by calling jl.yield() periodically from Python code, allowing the Julia scheduler to run.

When working at the Python REPL, you may call juliacall.interactive() which will allow Julia async code to run while the prompt is showing. This will allow interactive plots such as Makie to work.

If you can disable threading somehow (it would be slower), just to test, and you get rid of your error, then threading is the most likely, expected, cause of your problem. You would be safe to run your code that way, if you can tolerate losing out on threading, at least until you migrate to a better solution.

How can I disable threading in Flask?

I had heard of Flask, but not Celery. The latter seems to be to handle long-running task from Flask, i.e. not in the same thread, likely spawns its own. You may need to disable Celery or its threading (which seems its main point). Possibly if Celery is just used on the Python side, not for calling Julia then maybe you don’t need to interfere with it (despite the error message pointing to it). But I very much doubt you get away with keeping Celery unchanged, and it might be code dependant, possibly ok for some of your code, but not other Celery using code.