I work at a very small (3 devs) startup. Our backend is Python / Django. We have some computation-intensive code for demand estimation which we’ve written in Julia, and we want to integrate that code into our Django backend and wrap/expose it via endpoint.
So, in our Django project we have an app
demand_estimation which lives alongside the usual suspects of
docs, etc. Inside the
demand_estimation directory there is:
DemandEstimation.jl, which defines module
DemandEstimationand contains the core logic;
demand_estimation.py, which wraps the Julia-side core logic with pyjulia (and defines endpoints);
__init__.py, which looks like:
# set up Julia sans precompilation (necessary for conda python) from julia.api import Julia Julia(compiled_modules=False) # load julia demand estimation code from julia import Pkg Pkg.activate(".") # django backend environment from julia import Main Main.include("demand_estimation/DemandEstimation.jl") from julia.Main import DemandEstimation as DE # noqa
demand_estimation.py we have
from . import DE and make calls like
DE.important_function(args). This works well…in tests.
However, when we actually try to run a debug/test/development server with Django’s
runserver command, there are segfaults left and right. In particular, we’ve traced the segfaults to the following code in
from . import DE # DE is DemandEstimation julia module defined in the __init__.py def call_something_in_julia(arg): func_to_call = DE.important_func # this line causes a segfault return func_to_call(arg)
To reiterate, a Django test that hits
call_something_in_julia will pass without error; hitting a running server’s endpoint that in turn calls
call_something_in_julia causes a segfault.
Any idea on what might be causing this segfault, how to mitigate it, and/or why it happens only when running a server and not in tests? We’re pretty much at wit’s end and considering just abandoning the backend Julia, which is a shame for lots of reasons, not least because I greatly prefer writing Julia over Python!