# Julia crashes inside @threads with MPI

**URL:** <https://discourse.julialang.org/t/julia-crashes-inside-threads-with-mpi/52400>\
**Category:** Julia at Scale\
**Tags:** multithreading, mpi\
**Created:** [December 26, 2020, 12:25pm UTC](https://discourse.julialang.org/t/julia-crashes-inside-threads-with-mpi/52400 "2020-12-26T12:25:03Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jae-Mo\_Lihm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jae-mo_lihm/32/20145_2.png) [@Jae-Mo\_Lihm](https://discourse.julialang.org/u/Jae-Mo_Lihm)\
**Post date:** [December 26, 2020, 12:25pm UTC](https://discourse.julialang.org/t/julia-crashes-inside-threads-with-mpi/52400/1 "2020-12-26T12:25:03Z")

</div>

Hi, I am trying to implement a program with hybrid MPI + Threads parallelization.

In the following example, if I comment out all MPI-related lines, the code runs without any problem. However, if I run the code with the MPI lines, Julia crashes in a probabilistic manner (crash probability around 50%).

```julia
# mwe.jl
using MPI
using Base.Threads

MPI.Init_thread(MPI.THREAD_FUNNELED)
world_comm = MPI.COMM_WORLD

struct MyStruct
    v::Vector{Vector{Float64}}
end

mystruct = MyStruct([[1.0] for i=1:nthreads()])

println("With @threads")
Threads.@threads :static for i in 1:nthreads()
    println(mystruct.v[threadid()])
end

MPI.Finalize()

```

Run `julia -t 2 mwe.jl`, output:

```julia
With @threads

signal (11): Segmentation fault
in expression starting at /home/jmlim/julia_epw/EPW.jl/running/mwe.jl:16
jl_mutex_wait at /buildworker/worker/package_linux64/build/src/locks.h:37 [inlined]
jl_mutex_lock at /buildworker/worker/package_linux64/build/src/locks.h:94
jl_generate_fptr at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:272
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1964
jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1919 [inlined]
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2224 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
println at ./coreio.jl:4
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2231 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
macro expansion at /home/jmlim/julia_epw/EPW.jl/running/mwe.jl:17 [inlined]
#3#threadsfor_fun at ./threadingconstructs.jl:81
#3#threadsfor_fun at ./threadingconstructs.jl:48
unknown function (ip: 0x2ab50928062c)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2231 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:705
unknown function (ip: (nil))
Allocations: 780384 (Pool: 780097; Big: 287); GC: 1
Segmentation fault (core dumped)

```

`julia -t 1 mwe.jl` output is okay:

```julia
With @threads
[1.0]

```

Another observation is that if I use `v = [[1.0] for i=1:nthreads()]` as an independent variable, not as a field of a struct `mystruct.v`, Julia does not crash.

What is the reason for this crash?

I am using Intel MPI Version 2019 on Linux (CentOS 7). I downloaded `Julia 1.5.3` binary. I tested two MPI.jl verisions: `v0.14.3`, `v0.16.1` (most recent) and both crashes.

```julia
julia> MPI.identify_implementation()
(MPI.IntelMPI, v"2019.0.0")

```

(In the real program, the `v::Vector{Vector{Float64}}` field is used as a pre-allocated buffer, one for each threads.)

---

<div class="post-metadata">

**Author:** ![Jae-Mo\_Lihm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jae-mo_lihm/32/20145_2.png) [@Jae-Mo\_Lihm](https://discourse.julialang.org/u/Jae-Mo_Lihm)\
**Post date:** [December 27, 2020, 4:22am UTC](https://discourse.julialang.org/t/julia-crashes-inside-threads-with-mpi/52400/2 "2020-12-27T04:22:50Z")

</div>

It seems that the issue comes from the system MPI.  
By using the automatically downloaded MPICH, the problem does not occur.

---

<div class="post-metadata">

**Author:** ![lyonsquark](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lyonsquark/32/4483_2.png) [@lyonsquark](https://discourse.julialang.org/u/lyonsquark)\
**Post date:** [December 28, 2020, 6:56am UTC](https://discourse.julialang.org/t/julia-crashes-inside-threads-with-mpi/52400/3 "2020-12-28T06:56:10Z")

</div>

If you are calling `MPI.Init`, you typically have to run Julia with `mpirun`, `mpiexec` or `srun` (depends on your system and the MPI variant). For example

```bash
# Run with one MPI rank
mpiexec -n 1 julia -t 2 mwe.jl

```

---

<div class="post-metadata">

**Author:** ![Jae-Mo\_Lihm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jae-mo_lihm/32/20145_2.png) [@Jae-Mo\_Lihm](https://discourse.julialang.org/u/Jae-Mo_Lihm)\
**Post date:** [December 28, 2020, 2:20pm UTC](https://discourse.julialang.org/t/julia-crashes-inside-threads-with-mpi/52400/4 "2020-12-28T14:20:54Z")

</div>

Thank you for the reply. And thank you for the information about the `mpirun` and `mpiexec` command.  
Unfortunately, the crash persists when using `mpirun` or `mpiexec`.

Since the crash does not happen for Julia-provided MPICH, I suspect that the system MPI is the reason for the crash.

---

<div class="post-metadata">

**Author:** ![fverdugo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fverdugo/32/9446_2.png) [@fverdugo](https://discourse.julialang.org/u/fverdugo)\
**Post date:** [December 28, 2020, 5:26pm UTC](https://discourse.julialang.org/t/julia-crashes-inside-threads-with-mpi/52400/5 "2020-12-28T17:26:37Z")

</div>

Hi @Jae-Mo_Lihm  
Take a look at this section of the MPI.jl manual:  
[https://juliaparallel.github.io/MPI.jl/stable/knownissues/#Multi-threading-and-signal-handling](https://juliaparallel.github.io/MPI.jl/stable/knownissues/#Multi-threading-and-signal-handling)

---

<div class="post-metadata">

**Author:** ![Jae-Mo\_Lihm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jae-mo_lihm/32/20145_2.png) [@Jae-Mo\_Lihm](https://discourse.julialang.org/u/Jae-Mo_Lihm)\
**Post date:** [December 29, 2020, 7:41am UTC](https://discourse.julialang.org/t/julia-crashes-inside-threads-with-mpi/52400/6 "2020-12-29T07:41:52Z")

</div>

Hi @fverdugo  
Thank you for the pointer! My case indeed seems very related to the issue [https://github.com/JuliaParallel/MPI.jl/issues/337](https://github.com/JuliaParallel/MPI.jl/issues/337).

The error seems to be fixed for a recent version of OpenMPI. I am using intel MPI and the error occurs. Calling `export UCX_ERROR_SIGNALS="SIGILL,SIGBUS,SIGFPE"` did not fix the crash.

```julia
$ mpirun --version
Intel(R) MPI Library for Linux* OS, Version 2019 Update 9 Build 20200923 (id: abd58e492)
Copyright 2003-2020, Intel Corporation.

```