# Using LinearMap with MPI.jl
**URL:** https://discourse.julialang.org/t/using-linearmap-with-mpi-jl/91464
**Category:** Performance
**Tags:** linearalgebra, mpi
**Created:** [December 9, 2022, 10:40am UTC](https://discourse.julialang.org/t/using-linearmap-with-mpi-jl/91464 "2022-12-09T10:40:51Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@devanshu](https://discourse.julialang.org/u/devanshu)
#### Post date: [December 9, 2022, 10:40am UTC](https://discourse.julialang.org/t/using-linearmap-with-mpi-jl/91464/1 "2022-12-09T10:40:51Z")
Hi,
I am finding difficulty in using `LinearMaps` while also trying to parallelize the code using `MPI.jl`.
Actually, I am using `ArnoldiMethod.jl` in order to implement _shift-invert diagonalization_ technique. The minimal code is this
```julia
A = Matrix(A) # because A belongs to SparseArrays
F = lu(A - s * I)
A_lmap = LinearMap{eltype(A)}((y, x) -> ldiv!(y, F, x), size(A, 1), ismutating=true)
decomp, = partialschur(A_lmap, nev=n, tol=1e-5, restarts=100, which=LM())
_, X = partialeigen(decomp)
return X
```
My code terminates at line three with the error
```julia
ERROR: LoadError: MethodError: objects of type Int64 are not callable
Maybe you forgot to use an operator such as *, ^, %, / etc. ?
```
Without MPI, the code works fine. I am not able to figure out where is it going wrong while using MPI.
Thanks for the help.
---
### Author:  [@jishnub](https://discourse.julialang.org/u/jishnub)
#### Post date: [December 9, 2022, 11:43am UTC](https://discourse.julialang.org/t/using-linearmap-with-mpi-jl/91464/2 "2022-12-09T11:43:16Z")
What’s the MPI part here? Could you also post the full error stracktrace?
---
### Author:  [@devanshu](https://discourse.julialang.org/u/devanshu)
#### Post date: [December 9, 2022, 12:12pm UTC](https://discourse.julialang.org/t/using-linearmap-with-mpi-jl/91464/3 "2022-12-09T12:12:08Z")
The whole code is a bit big that’s why I didn’t put it here. I was just expecting to know if there are any known caveats to using `LinearMap` etc. with MPI. However, here is the minimal code to give you an idea of what I am trying to do, although it’s not complete.
```julia
function shiftInverse(A, s, n)
A = Matrix(A)
F = lu(A - s * I)
A_lmap = LinearMap{eltype(A)}((y, x) -> ldiv!(y, F, x), size(A, 1), ismutating=true)
decomp, = partialschur(A_lmap, nev=n, tol=1e-5, restarts=100, which=LM())
_, X = partialeigen(decomp)
return X
end
h_arr = round.(LinRange(0.1, 5, 10), digits=3)
J = 1.0; ϵ = 0.5; nphi = 1
l = 4
MPI.Init()
comm = MPI.COMM_WORLD
size = MPI.Comm_size(comm)
rank = MPI.Comm_rank(comm)
magLenNodes = div(length(h_arr), size-1)
if rank == 0
for z in 1:size-1
send_msg = h_arr[magLenNodes*(z-1)+1:magLenNodes*z]
MPI.send(send_msg, comm; dest=z)
end
else
magRec = MPI.recv(comm; source=0)
print("received on rank = $rank\n")
# flush(stdout)
for h in magRec
H = buildHNonRnd(η, l, basis, J) + buildHRnd(h, basis, η, l)
e0, = eigs(H; nev=1, which=:SM, ritzvec=false)
emax, = eigs(H; nev=1, which=:LM, ritzvec=false)
e = (e0[1] - emax[1]) * ϵ + emax[1]
ϕ = shiftInverse(H, e, nphi)
end
end
MPI.Barrier(comm)
```
The complete code would require more functions to be added, which I think is irrelevant. The code terminates at line 3 of the function `shiftInverse` above. Here is the full stacktrace
```julia
ERROR: LoadError: MethodError: objects of type Int64 are not callable
Maybe you forgot to use an operator such as *, ^, %, / etc. ?
Stacktrace:
[1] shiftInverse(A::SparseMatrixCSC{Float64, Int64}, s::Float64, n::Int64)
@ Main /scratch/20ph92r03/juliafiles/mpiTest/mblPTmpi.jl:161
[2] entDynamicsMbl(mag::Vector{Float64}, l::Int64, J::Float64, ϵ::Float64, nphi::Int64, ens::Int64, avgEE::Dict{Any, Any}, stdEE::Dict{Any, Any})
@ Main /scratch/20ph92r03/juliafiles/mpiTest/mblPTmpi.jl:216
[3] top-level scope
@ /scratch/20ph92r03/juliafiles/mpiTest/mblPTmpi.jl:276
in expression starting at /scratch/20ph92r03/juliafiles/mpiTest/mblPTmpi.jl:263
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 0 PID 3502 RUNNING AT cn106
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
Thank you for your time!
```
---
### Author:  [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)
#### Post date: [December 9, 2022, 2:58pm UTC](https://discourse.julialang.org/t/using-linearmap-with-mpi-jl/91464/4 "2022-12-09T14:58:17Z")
I suspect that the issue is that you globally define `size = MPI.Comm_size(comm)` which is an `Int64`. This overshadows the `size` function (i.e. `Base.size`). In `shiftInverse` you then call `size(A, 1)` which doesn’t work because objects of type `Int64` (the global `size` object) isn’t callable.
In principle, you could solve this by explicitly using `Base.size(A, 1)` in `shiftInverse` but I strongly recommended to get rid of these global variables in the first place. The simple solution is to put everything in a function. Essentially there shouldn’t be any code that doesn’t live in a function.
For example
```julia
function shiftInverse(A, s, n)
A = Matrix(A)
F = lu(A - s * I)
A_lmap = LinearMap{eltype(A)}((y, x) -> ldiv!(y, F, x), size(A, 1), ismutating=true)
decomp, = partialschur(A_lmap, nev=n, tol=1e-5, restarts=100, which=LM())
_, X = partialeigen(decomp)
return X
end
function main()
h_arr = round.(LinRange(0.1, 5, 10), digits=3)
J = 1.0; ϵ = 0.5; nphi = 1
l = 4
MPI.Init()
comm = MPI.COMM_WORLD
size = MPI.Comm_size(comm)
rank = MPI.Comm_rank(comm)
magLenNodes = div(length(h_arr), size-1)
if rank == 0
for z in 1:size-1
send_msg = h_arr[magLenNodes*(z-1)+1:magLenNodes*z]
MPI.send(send_msg, comm; dest=z)
end
else
magRec = MPI.recv(comm; source=0)
print("received on rank = $rank\n")
# flush(stdout)
for h in magRec
H = buildHNonRnd(η, l, basis, J) + buildHRnd(h, basis, η, l)
e0, = eigs(H; nev=1, which=:SM, ritzvec=false)
emax, = eigs(H; nev=1, which=:LM, ritzvec=false)
e = (e0[1] - emax[1]) * ϵ + emax[1]
ϕ = shiftInverse(H, e, nphi)
end
end
MPI.Barrier(comm)
end
main() # this runs your computation
```
---
### Author:  [@devanshu](https://discourse.julialang.org/u/devanshu)
#### Post date: [December 10, 2022, 5:49am UTC](https://discourse.julialang.org/t/using-linearmap-with-mpi-jl/91464/5 "2022-12-10T05:49:29Z")
Yes, you are right. It was really silly. It works now. Thank you!
---
### Author:  [@devanshu](https://discourse.julialang.org/u/devanshu)
#### Post date: [December 11, 2022, 6:34am UTC](https://discourse.julialang.org/t/using-linearmap-with-mpi-jl/91464/6 "2022-12-11T06:34:24Z")
Hi,
I have some issues while using multi-threading with MPI. For multi-threading, I am using the `Floops` package.
```julia
function func(mag, l, J, ϵ, nphi, ens)
basis = buildBasis(l)
N = 2^l; η = binomial(l, Int(l/2))
# avgEE = similar(mag)
stdEE = similar(mag)
for (i,h) in enumerate(mag)
H_init = buildHNonRnd(η, l, basis, J) + buildHRnd(h, basis, η, l)
e0, = eigs(H_init; nev=1, which=:SM, ritzvec=false)
emax, = eigs(H_init; nev=1, which=:LM, ritzvec=false)
e = (e0[1] - emax[1]) * ϵ + emax[1]
H_0 = buildHNonRnd(η, l, basis, J)
ee = Vector{Float64}(undef, ens)
@floop for m in 1:ens
H = H_0 + buildHRnd(h, basis, η, l)
ϕ = shiftInverse(H, e, nphi)
ee[m] = getEE(ϕ, N, η, l, nphi)
end
# avgEE[i] = mean(ee)
stdEE[i] = std(ee)
print("completed h = $h \n")
flush(stdout)
end
return stdEE
end
```
Also, I have slightly changed the MPI part.
```julia
function main()
L = 4
sampleSize = 500
h_arr = round.(LinRange(0.1, 8, 30), digits=3)
J = 1.0; ϵ = 0.5; nphi = 5
MPI.Init()
comm = MPI.COMM_WORLD
size = MPI.Comm_size(comm)
rank = MPI.Comm_rank(comm)
magLenNodes = div(length(h_arr), size)
send_buf = zeros(magLenNodes)
magRec = MPI.Scatter!(MPI.UBuffer(h_arr, magLenNodes), send_buf, 0, comm)
magRec = func(send_buf, L, J, ϵ, nphi, sampleSize)
recv_msg = MPI.Gather(magRec, 0, comm)
if rank == 0
print("completed, $recv_msg \n")
end
# end
MPI.Barrier(comm)
end
```
I run the code with `JULIA_NUM_THREADS=4` and `mpiexec -n 2 julia example.jl`, and got this error:
```julia
signal (11): Segmentation fault
signal (11): Segmentation fault
in expression starting at /scratch/20ph92r03/juliafiles/proj_2/mblPTmpi.jl:314
in expression starting at /scratch/20ph92r03/juliafiles/proj_2/mblPTmpi.jl:314
jl_gc_alloc at /home/20ph92r03/julia-1.8.0/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
ijl_alloc_array_1d at /home/20ph92r03/julia-1.8.0/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
Array at ./boot.jl:459 [inlined]
Array at ./boot.jl:468 [inlined]
similar at ./array.jl:378 [inlined]
similar at ./subarray.jl:65 [inlined]
similar at ./abstractarray.jl:797 [inlined]
* at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/LinearAlgebra/src/matmul.jl:101 [inlined]
orthogonalize! at /home/20ph92r03/.julia/packages/ArnoldiMethod/JdEiw/src/expansion.jl:89
iterate_arnoldi! at /home/20ph92r03/.julia/packages/ArnoldiMethod/JdEiw/src/expansion.jl:123
_partialschur at /home/20ph92r03/.julia/packages/ArnoldiMethod/JdEiw/src/run.jl:185
jl_gc_state_set at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/julia_threads.h:340 [inlined]
#partialschur#1 at /home/20ph92r03/.julia/packages/ArnoldiMethod/JdEiw/src/run.jl:106
maybe_collect at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/julia_threads.h:333 [inlined]
jl_gc_pool_alloc_inner at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gc.c:1247 [inlined]
partialschur##kw at /home/20ph92r03/.julia/packages/ArnoldiMethod/JdEiw/src/run.jl:94 [inlined]
shiftInverse at /scratch/20ph92r03/juliafiles/proj_2/mblPTmpi.jl:164
jl_gc_pool_alloc_noinline at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gc.c:1306 [inlined]
macro expansion at /scratch/20ph92r03/juliafiles/proj_2/mblPTmpi.jl:217 [inlined]
__##reducing_function#313 at /home/20ph92r03/.julia/packages/FLoops/3ZEuy/src/reduce.jl:817 [inlined]
AdjoinIdentity at /home/20ph92r03/.julia/packages/InitialValues/OWP8V/src/InitialValues.jl:306
jl_gc_alloc_ at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/julia_internal.h:369 [inlined]
jl_gc_alloc at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gc.c:3371
next at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/combinators.jl:290 [inlined]
next at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/core.jl:289 [inlined]
macro expansion at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/core.jl:181 [inlined]
macro expansion at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/processes.jl:199 [inlined]
macro expansion at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/simd.jl:41 [inlined]
_foldl_linear_bulk at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/processes.jl:198 [inlined]
macro expansion at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/processes.jl:192 [inlined]
macro expansion at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/basics.jl:115 [inlined]
_foldl_array at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/processes.jl:188 [inlined]
__foldl__ at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/processes.jl:182 [inlined]
foldl_basecase at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/processes.jl:365 [inlined]
_reduce_basecase at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/threading_utils.jl:56
_reduce at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/reduce.jl:150
#167 at ./threadingconstructs.jl:258
unknown function (ip: 0x2ad40bc67caf)
_new_array_ at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/array.c:134 [inlined]
_new_array at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/array.c:198 [inlined]
ijl_alloc_array_1d at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/array.c:436
_jl_invoke at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2549
Array at ./boot.jl:459 [inlined]
Array at ./boot.jl:468 [inlined]
similar at ./array.jl:378 [inlined]
similar at ./abstractarray.jl:797 [inlined]
getrf! at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/LinearAlgebra/src/lapack.jl:563
#lu!#172 at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/LinearAlgebra/src/lu.jl:81 [inlined]
lu!##kw at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/LinearAlgebra/src/lu.jl:80 [inlined]
#lu#179 at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/LinearAlgebra/src/lu.jl:279 [inlined]
lu at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/LinearAlgebra/src/lu.jl:278 [inlined]
lu at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/LinearAlgebra/src/lu.jl:278 [inlined]
shiftInverse at /scratch/20ph92r03/juliafiles/proj_2/mblPTmpi.jl:160
macro expansion at /scratch/20ph92r03/juliafiles/proj_2/mblPTmpi.jl:217 [inlined]
__##reducing_function#313 at /home/20ph92r03/.julia/packages/FLoops/3ZEuy/src/reduce.jl:817 [inlined]
AdjoinIdentity at /home/20ph92r03/.julia/packages/InitialValues/OWP8V/src/InitialValues.jl:306
next at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/combinators.jl:290 [inlined]
next at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/core.jl:289 [inlined]
macro expansion at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/core.jl:181 [inlined]
macro expansion at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/processes.jl:199 [inlined]
macro expansion at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/simd.jl:41 [inlined]
_foldl_linear_bulk at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/processes.jl:198 [inlined]
macro expansion at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/processes.jl:192 [inlined]
macro expansion at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/basics.jl:115 [inlined]
_foldl_array at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/processes.jl:188 [inlined]
__foldl__ at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/processes.jl:182 [inlined]
foldl_basecase at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/processes.jl:365 [inlined]
_reduce_basecase at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/threading_utils.jl:56
_reduce at /home/20ph92r03/.julia/packages/Transducers/HBMTc/src/reduce.jl:150
#167 at ./threadingconstructs.jl:258
unknown function (ip: 0x2ad40bc67caf)
_jl_invoke at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2549
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 1 PID 53989 RUNNING AT cn039
= KILLED BY SIGNAL: 11 (Segmentation fault)
===================================================================================
```
I read that using Multi-threading has known issues in the `MPI.jl` documentation ([Known issues · MPI.jl](https://juliaparallel.org/MPI.jl/stable/knownissues/#Multi-threading-and-signal-handling)). However, the fix prescribed there doesn’t work for me. I even tried setting the environment variable` UCX_ERROR_SIGNALS=""`, but it crashes.
Can you please suggest something which I can try? It works without multi-threading though!