# Addprocs() fails with IOError on SLURM

**URL:** <https://discourse.julialang.org/t/addprocs-fails-with-ioerror-on-slurm/37951>\
**Category:** Julia at Scale\
**Tags:** parallel, distributed\
**Created:** [April 21, 2020, 8:55am UTC](https://discourse.julialang.org/t/addprocs-fails-with-ioerror-on-slurm/37951 "2020-04-21T08:55:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dkiese](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dkiese](https://discourse.julialang.org/u/dkiese)\
**Post date:** [April 21, 2020, 8:55am UTC](https://discourse.julialang.org/t/addprocs-fails-with-ioerror-on-slurm/37951/1 "2020-04-21T08:55:54Z")

</div>

Hello everyone,

I am currently trying to run hybrid shared-distributed memory code on a large computing cluster, which is configured with SLURM. My code is roughly structured like this

```julia
using Distributed
using ClusterManagers

addprocs(SlurmManager(number_of_nodes), topology = :master_worker)
include(joinpath(@ __DIR__ , "src.jl"))
main()

```

Here, “src.jl” is the code that I want to use which includes several packages (e.g. HDF5, ParallelDataTransfer … ) and subfiles first on the master process and then shares with the workers using `@everywhere`. `main()` is the function to be run from that code. Oddly, when I submit jobs I sometimes get `IOError: unlink: no such file or directory (ENOENT)` with the stack trace pointing me towards the `addprocs` line and the job is terminated. If I resubmit however, it sometimes works flawlessly. Is there any paradigm I am missing when using ClusterManagers on a SLURM system?

EDIT:

The full error message is

```julia
ERROR: LoadError: TaskFailedException:
IOError: unlink: no such file or directory (ENOENT)
Stacktrace:
 [1] uv_error at ./libuv.jl:97 [inlined]
 [2] unlink(::String) at ./file.jl:781
 [3] #rm#9(::Bool, ::Bool, ::typeof(rm), ::String) at ./file.jl:261
 [4] rm at ./file.jl:253 [inlined]
 [5] iterate at ./generator.jl:47 [inlined]
 [6] _collect(::Array{String,1}, ::Base.Generator{Array{String,1},typeof(rm)}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:635
 [7] collect_similar at ./array.jl:564 [inlined]
 [8] map at ./abstractarray.jl:2073 [inlined]
 [9] launch(::SlurmManager, ::Dict{Symbol,Any}, ::Array{WorkerConfig,1}, ::Base.GenericCondition{Base.AlwaysLockedST}) at /p/scratch/chku27/kiese1/julia_packages/packages/ClusterManagers/7pPEP/src/slurm.jl:39
 [10] (::Distributed.var"#41#44"{SlurmManager,Dict{Symbol,Any},Array{WorkerConfig,1},Base.GenericCondition{Base.AlwaysLockedST}})() at ./task.jl:333
Stacktrace:
 [1] wait at ./task.jl:251 [inlined]
 [2] #addprocs_locked#40(::Base.Iterators.Pairs{Symbol,Symbol,Tuple{Symbol},NamedTuple{(:topology,),Tuple{Symbol}}}, ::typeof(Distributed.addprocs_locked), ::SlurmManager) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:494
 [3] #addprocs_locked at ./none:0 [inlined]
 [4] #addprocs#39(::Base.Iterators.Pairs{Symbol,Symbol,Tuple{Symbol},NamedTuple{(:topology,),Tuple{Symbol}}}, ::typeof(addprocs), ::SlurmManager) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/cluster.jl:441
 [5] (::Distributed.var"#kw##addprocs")(::NamedTuple{(:topology,),Tuple{Symbol}}, ::typeof(addprocs), ::SlurmManager) at ./none:0
 [6] top-level scope at /p/scratch/chku27/kiese1/fcc_ml/num_param/lattice/fcc_L6_W44_N125_1l.jl:5
 [7] include at ./boot.jl:328 [inlined]
 [8] include_relative(::Module, ::String) at ./loading.jl:1105
 [9] include(::Module, ::String) at ./Base.jl:31
 [10] exec_options(::Base.JLOptions) at ./client.jl:287
 [11] _start() at ./client.jl:460

```

---

<div class="post-metadata">

**Author:** ![dmolina](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmolina/32/5246_2.png) [@dmolina](https://discourse.julialang.org/u/dmolina)\
**Post date:** [April 21, 2020, 10:53am UTC](https://discourse.julialang.org/t/addprocs-fails-with-ioerror-on-slurm/37951/2 "2020-04-21T10:53:38Z")

</div>

I am not completely sure, because these days I have not access to my cluster, but in SLURM usually you have to put the absolute path when you work with files, because the workers have initially a different relative path.

I hope this help you.

---

<div class="post-metadata">

**Author:** ![dkiese](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@dkiese](https://discourse.julialang.org/u/dkiese)\
**Post date:** [April 21, 2020, 11:19am UTC](https://discourse.julialang.org/t/addprocs-fails-with-ioerror-on-slurm/37951/3 "2020-04-21T11:19:57Z")

</div>

Thanks for the reply. That is actually the way I do it. I have edited my post accordingly. Anyways, the error seems to happen already when the ClusterManager tries to set up the workers, so there should not have been any file loading yet.