Not able to run Julia through Flask-Celery

We are working on SaaS based platform implemented using Flask and Celery.
In the backend, we have some part of the implementation written in julia.
We integrated julia using a python library called julia==0.6.0
In the flask we are able to run the julia code.
However we are not able to run the same code inside the celery task. The error that we get is

celery_worker_1 | signal (4): Illegal instruction
celery_worker_1 | in expression starting at none:2
celery_worker_1 | unknown function (ip: 0xffff727d6cec)
celery_worker_1 | _jl_invoke at /cache/build/default-awsarm-5/julialang/julia-release-1-dot-8/src/gf.c:2365 [inlined]
celery_worker_1 | ijl_apply_generic at /cache/build/default-awsarm-5/julialang/julia-release-1-dot-8/src/gf.c:2547
celery_worker_1 | Allocations: 3462340 (Pool: 3459082; Big: 3258); GC: 4
celery_worker_1 | [2022-12-18

We are using julia v1.8.3 (even we have tried v1.9.0-alpha1 but we are getting the same error).

Here’s the repo for a reproducible example: GitHub - p-oorja/p-oorja: Config files for my GitHub profile.

Welcome to the Julia discourse!

Has anybody got solution to this…

I suspect the users of a Celery / Python forum might be more equipped to answer your question.

We integrated julia using a python library called julia==0.6.0

It seems you’re using PyJulia package, but note its docs, under “Limitations” (your error is likely a race condition because of the given limitation of the software you use):

No threading support

PyJulia cannot be used in different threads since libjulia is not thread safe. However, you can use multiple threads within Julia. For example, start IPython by JULIA_NUM_THREADS=4 ipython and then run:

In [1]: %load_ext julia.magic Initializing Julia runtime. This may take some time… In [2]: %%julia …: a = zeros(10) …: Threads.@threads for i = 1:10 …: a[i] = Threads.threadid() …: end …: a Out[3]: array([1., 1., 1., 2., 2., 2., 3., 3., 4., 4.])

PyJulia does not release GIL

PyJulia does not release the Global Interpreter Lock (GIL) while calling Julia functions since PyCall expects the GIL to be acquired always. It means that Python code and Julia code cannot run in parallel.

I recognize the error you have, as something coming from Julia without the “celery_worker_1 |” prefix, which I believe the Python side added somehow. I suppose that’s celery worker 1 (of many), i.e. a thread?

You can use all Python libraries from Julia (i.e. Julia is your main language) through PythonCall.jl (or older PyCall.jl). With it or PyJulia, you can also use all Julia libraries from Python. In both cases some limitations may apply:

When you e.g. call Julia from Python, the library calling may also be a framework. Frameworks call you (“the Hollywood principle” of frameworks), so they must run in the “top” language, as you are doing with Flask.

You may want to consider some popular Julia (web) framework such as Genie.jl as a replacement for Python’s Flask (and replace all Python framework stuff), or e.g. this alternative:

Because serving web pages means a lot of requests come in and you want to serve them as fast as possible, meaning in parallel (“asynchronous”), often done with threading, something those interop libraries may not support. Unless possible if you jump through some hoops. Something you can avoid if using a full Julia (or full Python) solution.

Asynchronous Julia code (including Makie)

Asynchronous Julia code will not normally run while Python is executing, unless it is in a separate thread.

This can be fixed by calling jl.yield() periodically from Python code, allowing the Julia scheduler to run.

When working at the Python REPL, you may call juliacall.interactive() which will allow Julia async code to run while the prompt is showing. This will allow interactive plots such as Makie to work.

If you can disable threading somehow (it would be slower), just to test, and you get rid of your error, then threading is the most likely, expected, cause of your problem. You would be safe to run your code that way, if you can tolerate losing out on threading, at least until you migrate to a better solution.

I had heard of Flask, but not Celery. The latter seems to be to handle long-running task from Flask, i.e. not in the same thread, likely spawns its own. You may need to disable Celery or its threading (which seems its main point). Possibly if Celery is just used on the Python side, not for calling Julia then maybe you don’t need to interfere with it (despite the error message pointing to it). But I very much doubt you get away with keeping Celery unchanged, and it might be code dependant, possibly ok for some of your code, but not other Celery using code.

1 Like

Thank you for mentioning my package @Palli .

To answer the initial question of @shwetank_verma, I recommend configuring backend in DanceJL, looking at limited endpoints from your code. You can easily add a simple HTTP Bearer check if necessary.

Also there is no equivalent of a Celery library in Julia. So why not use something such as AWS MQ, with GitHub - rweilbacher/MQTT.jl: An asynchronous MQTT client library for julia. That would be fastest way moving forward, in my opinion.

1 Like