# Pyjulia, Django, and segfaults

**URL:** <https://discourse.julialang.org/t/pyjulia-django-and-segfaults/25313>\
**Category:** General Usage\
**Created:** [June 14, 2019, 4:22pm UTC](https://discourse.julialang.org/t/pyjulia-django-and-segfaults/25313 "2019-06-14T16:22:40Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![evanfields](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/evanfields/32/1744_2.png) [@evanfields](https://discourse.julialang.org/u/evanfields)\
**Post date:** [June 14, 2019, 4:22pm UTC](https://discourse.julialang.org/t/pyjulia-django-and-segfaults/25313/1 "2019-06-14T16:22:40Z")

</div>

I work at a very small (3 devs) startup. Our backend is Python / Django. We have some computation-intensive code for demand estimation which we’ve written in Julia, and we want to integrate that code into our Django backend and wrap/expose it via endpoint.

So, in our Django project we have an app `demand_estimation` which lives alongside the usual suspects of `db`, `util`, `docs`, etc. Inside the `demand_estimation` directory there is:

- `DemandEstimation.jl`, which defines module `DemandEstimation` and contains the core logic;
- `demand_estimation.py`, which wraps the Julia-side core logic with pyjulia (and defines endpoints);
- ` __init__.py`, which looks like:

```julia
# set up Julia sans precompilation (necessary for conda python)
from julia.api import Julia

Julia(compiled_modules=False)
# load julia demand estimation code
from julia import Pkg

Pkg.activate(".") # django backend environment
from julia import Main

Main.include("demand_estimation/DemandEstimation.jl")

from julia.Main import DemandEstimation as DE # noqa

```

Then in `demand_estimation.py` we have `from . import DE` and make calls like `DE.important_function(args)`. This works well…in tests.

However, when we actually try to run a debug/test/development server with Django’s `runserver` command, there are segfaults left and right. In particular, we’ve traced the segfaults to the following code in `demand_estimation.py`:

```julia
from . import DE # DE is DemandEstimation julia module defined in the __init__.py

def call_something_in_julia(arg):
    func_to_call = DE.important_func # this line causes a segfault
    return func_to_call(arg)

```

To reiterate, a Django test that hits `call_something_in_julia` will pass without error; hitting a running server’s endpoint that in turn calls `call_something_in_julia` causes a segfault.

Any idea on what might be causing this segfault, how to mitigate it, and/or why it happens only when running a server and not in tests? We’re pretty much at wit’s end and considering just abandoning the backend Julia, which is a shame for lots of reasons, not least because I greatly prefer writing Julia over Python!

---

<div class="post-metadata">

**Author:** ![evanfields](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/evanfields/32/1744_2.png) [@evanfields](https://discourse.julialang.org/u/evanfields)\
**Post date:** [June 14, 2019, 6:40pm UTC](https://discourse.julialang.org/t/pyjulia-django-and-segfaults/25313/2 "2019-06-14T18:40:32Z")

</div>

Responding to my own post in case someone searches with a similar problem. It _seems_ like we are running into trouble because libjulia is not thread-safe. Tentatively, running the Django test server with `--nothreading --noreload` eliminates the segfaults. Presumably the Django tests are single threaded.

---

<div class="post-metadata">

**Author:** ![nlaws](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nlaws/32/22205_2.png) [@nlaws](https://discourse.julialang.org/u/nlaws)\
**Post date:** [February 22, 2021, 4:26pm UTC](https://discourse.julialang.org/t/pyjulia-django-and-segfaults/25313/3 "2021-02-22T16:26:18Z")

</div>

Hi, we have a similar app set-up and are also seeing nebulous (periodic) segfaults: did you figure out how to avoid them in production? (We are using gunicorn in production).

---

<div class="post-metadata">

**Author:** ![evanfields](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/evanfields/32/1744_2.png) [@evanfields](https://discourse.julialang.org/u/evanfields)\
**Post date:** [February 22, 2021, 5:07pm UTC](https://discourse.julialang.org/t/pyjulia-django-and-segfaults/25313/4 "2021-02-22T17:07:12Z")

</div>

The switch to `--nothreading` seemed to fix the segfaults at the time. Shortly after this thread we switched to running the Julia code in its own microservice, so I’m not up to date on Django + Gunicord + Julia-python compatibility. Sorry!

---

<div class="post-metadata">

**Author:** ![nlaws](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nlaws/32/22205_2.png) [@nlaws](https://discourse.julialang.org/u/nlaws)\
**Post date:** [February 22, 2021, 6:01pm UTC](https://discourse.julialang.org/t/pyjulia-django-and-segfaults/25313/5 "2021-02-22T18:01:10Z")

</div>

Ok thank you @evanfields! This thread did get me on the right track: our issue seems to be also that [PyJulia is not thread safe](https://pyjulia.readthedocs.io/en/latest/limitations.html#no-threading-support) (and we are using Celery with Django to run some tasks in parallel that use PyJulia).

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [February 22, 2021, 6:08pm UTC](https://discourse.julialang.org/t/pyjulia-django-and-segfaults/25313/6 "2021-02-22T18:08:03Z")

</div>

Of course it depends on your architecture, but in the world of micorservices it could make sense to implement important julia function as a standalone server and communicate with it internally. This way you will be free of direct python/julia interactions.

---

<div class="post-metadata">

**Author:** ![lungben](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lungben/32/12314_2.png) [@lungben](https://discourse.julialang.org/u/lungben)\
**Post date:** [February 22, 2021, 6:15pm UTC](https://discourse.julialang.org/t/pyjulia-django-and-segfaults/25313/7 "2021-02-22T18:15:04Z")

</div>

Just speculating, but would it help to use locks on Python-side so that only one thread at a time calls Julia?

---

<div class="post-metadata">

**Author:** ![nlaws](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nlaws/32/22205_2.png) [@nlaws](https://discourse.julialang.org/u/nlaws)\
**Post date:** [February 22, 2021, 7:06pm UTC](https://discourse.julialang.org/t/pyjulia-django-and-segfaults/25313/8 "2021-02-22T19:06:48Z")

</div>

@Skoffer Yes I think that setting up our Julia processes as a microservice is the way to go. We are deploying in Kubernetes, so I guess the way to go would be to set up an API in a Julia container for the python container to call?

@lungben Yes I think that approach will also work. I’m thinking of multi-threading on the Julia side of our app instead of on the python side as we are now.

---

<div class="post-metadata">

**Author:** ![Skoffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/skoffer/32/378_2.png) [@Skoffer](https://discourse.julialang.org/u/Skoffer)\
**Post date:** [February 22, 2021, 7:52pm UTC](https://discourse.julialang.org/t/pyjulia-django-and-segfaults/25313/9 "2021-02-22T19:52:15Z")

</div>

Yes, you can go with [Pages.jl](https://github.com/EricForgy/Pages.jl) or with vanilla [HTTP.jl](https://github.com/JuliaWeb/HTTP.jl). For a simple internal API it wouldn’t make much difference I suppose.