# Using a machine file on a cluster but also propagating environment to the remote workers

**URL:** <https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948>\
**Category:** Julia at Scale\
**Tags:** question\
**Created:** [May 7, 2020, 11:38am UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948 "2020-05-07T11:38:17Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![HarrisonWilde](https://avatars.discourse-cdn.com/v4/letter/h/f07891/32.png) [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Post date:** [May 7, 2020, 11:38am UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/1 "2020-05-07T11:38:17Z")

</div>

Hi, I am trying to run some distributed code on a slurm cluster using an sbatch file as below:

```julia
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --tasks-per-node=28
#SBATCH --mem-per-cpu=4571
#SBATCH --time=24:00:00
#SBATCH -o slurm.%N.%j.out # STDOUT
#SBATCH -e slurm.%N.%j.err # STDERR

# Create the machine file for Julia
JULIA_MACHINEFILE=machinefile-${SLURM_JOB_ID}
srun bash -c hostname > $JULIA_MACHINEFILE
sed -i 's/^[[:alnum:]]*/&-ib/g' $JULIA_MACHINEFILE

module purge
module load Julia/1.4.1-linux-x86_64

julia --machine-file ${JULIA_MACHINEFILE} \
    src/creditcard/run.jl \
    --dataset uci_heart \
    --label target \
    --epsilon 6.0 \
    --folds 5 \
    --distributed

```

My code looks like:

```julia
using Pkg; Pkg.activate(".")
using Distributed
using ArgParse
using ForwardDiff
using LinearAlgebra
using CSV
using DataFrames
using AdvancedHMC
...

@everywhere begin
    using Pkg; Pkg.activate(".")
    using Distributed
    using ArgParse
    using ForwardDiff
    using LinearAlgebra
    using CSV
    using DataFrames
    using AdvancedHMC
...

MAIN BODY OF CODE

```

However, when I submit by batch job, it always crashes out in a way that suggests the environment is not being set properly:

```julia
ERROR: LoadError: On worker 2:
ArgumentError: Package ArgParse [c7e460c6-2fb9-53a9-8c5b-16f535851c63] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

_require at ./loading.jl:998
require at ./loading.jl:927
#1 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/Distributed.jl:78
#101 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:290
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:79
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:88
#94 at ./task.jl:358

...and 27 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:316
 [2] macro expansion at ./task.jl:335 [inlined]
 [3] _require_callback(::Base.PkgId) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/Distributed.jl:75
 [4] #invokelatest#1 at ./essentials.jl:712 [inlined]
 [5] invokelatest at ./essentials.jl:711 [inlined]
 [6] require(::Base.PkgId) at ./loading.jl:930
 [7] require(::Module, ::Symbol) at ./loading.jl:922
 [8] include(::Module, ::String) at ./Base.jl:377
 [9] exec_options(::Base.JLOptions) at ./client.jl:288
 [10] _start() at ./client.jl:484

```

I’d guess this is some problem with setting Julia paths etc across the remote workers, would appreciate some help with this.

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [May 7, 2020, 1:38pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/2 "2020-05-07T13:38:41Z")

</div>

What happens when you change

```julia
using Pkg; Pkg.activate(".")

```

to

```julia
using Pkg; Pkg.activate("."); Pkg.instantiate()

```

in both places?

---

<div class="post-metadata">

**Author:** ![orialb](https://avatars.discourse-cdn.com/v4/letter/o/65b543/32.png) [@orialb](https://discourse.julialang.org/u/orialb)\
**Post date:** [May 7, 2020, 1:48pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/3 "2020-05-07T13:48:32Z")

</div>

I’m not sure that it is related to your issue, but I found that I have to set the environment variable `JULIA_PROJECT` to `path/to/environment` in order for things to properly work on the SLURM cluster I’m working on when using multiple workers.  
This should make the `Pkg.activate(".")` call redundant.

---

<div class="post-metadata">

**Author:** ![HarrisonWilde](https://avatars.discourse-cdn.com/v4/letter/h/f07891/32.png) [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Post date:** [May 7, 2020, 1:50pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/4 "2020-05-07T13:50:55Z")

</div>

Have tried that, no change in the output unfortunately

---

<div class="post-metadata">

**Author:** ![HarrisonWilde](https://avatars.discourse-cdn.com/v4/letter/h/f07891/32.png) [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Post date:** [May 7, 2020, 1:51pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/5 "2020-05-07T13:51:36Z")

</div>

Interesting, how do you set that environment variable? Should I do `export JULIA_PROJECT path/to/env` on the login node or something else?

---

<div class="post-metadata">

**Author:** ![orialb](https://avatars.discourse-cdn.com/v4/letter/o/65b543/32.png) [@orialb](https://discourse.julialang.org/u/orialb)\
**Post date:** [May 7, 2020, 1:57pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/6 "2020-05-07T13:57:55Z")

</div>

I’m doing it as part of the sbatch script, it should be set on the compute node. But I’m not using machine files so maybe it’s different?  
I just start the processes from within the Julia script with

```julia
using ClusterManagers
addprocs(SlurmManager(nworkers))

```

The is also the `--project=path/to/environment` flag that you can try using instead, I don’t remember if I tried that on my cluster (there are some older discussions here about workers and environment activation)

---

<div class="post-metadata">

**Author:** ![orialb](https://avatars.discourse-cdn.com/v4/letter/o/65b543/32.png) [@orialb](https://discourse.julialang.org/u/orialb)\
**Post date:** [May 7, 2020, 2:01pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/7 "2020-05-07T14:01:28Z")

</div>

I’m by no means a Slurm expert, but why do you need the machine file in your script ? wouldn’t the process start by default on the compute node when you send this script to the queue via sbatch?

---

<div class="post-metadata">

**Author:** ![HarrisonWilde](https://avatars.discourse-cdn.com/v4/letter/h/f07891/32.png) [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Post date:** [May 7, 2020, 2:39pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/8 "2020-05-07T14:39:26Z")

</div>

The machine file as I understand creates more processes for each remote worker associated with the sbatch, this sbatch results in the same errors as before:

```julia
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --tasks-per-node=28
#SBATCH --mem-per-cpu=4571
#SBATCH --time=24:00:00
#SBATCH -o slurm.%N.%j.out # STDOUT
#SBATCH -e slurm.%N.%j.err # STDERR

# Create the machine file for Julia
JULIA_MACHINEFILE=machinefile-${SLURM_JOB_ID}
srun bash -c hostname > $JULIA_MACHINEFILE
sed -i 's/^[[:alnum:]]*/&-ib/g' $JULIA_MACHINEFILE

export JULIA_PROJECT=/home/dcs/csrxgb/julia_stuff/Project.toml

module purge
module load Julia/1.4.1-linux-x86_64

julia \
    --machine-file ${JULIA_MACHINEFILE} \
    src/creditcard/run.jl \
    --dataset uci_heart \
    --label target \
    --epsilon 6.0 \
    --folds 5 \
    --distributed

```

I will now try with your suggestion to use ClusterManagers

---

<div class="post-metadata">

**Author:** ![HarrisonWilde](https://avatars.discourse-cdn.com/v4/letter/h/f07891/32.png) [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Post date:** [May 7, 2020, 2:45pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/9 "2020-05-07T14:45:14Z")

</div>

There is something different at least:

```julia
[login1 julia_stuff]$ cat slurm.cnode43.250472.err
ERROR: LoadError: LoadError: UndefVarError: @spawnat not defined
Stacktrace:
 [1] top-level scope
 [2] include(::Module, ::String) at ./Base.jl:377
 [3] exec_options(::Base.JLOptions) at ./client.jl:288
 [4] _start() at ./client.jl:484
in expression starting at /gpfs/home/dcs/csrxgb/julia_stuff/src/creditcard/run.jl:5
in expression starting at /gpfs/home/dcs/csrxgb/julia_stuff/src/creditcard/run.jl:4

```

---

<div class="post-metadata">

**Author:** ![HarrisonWilde](https://avatars.discourse-cdn.com/v4/letter/h/f07891/32.png) [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Post date:** [May 7, 2020, 3:03pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/10 "2020-05-07T15:03:18Z")

</div>

Nevermind that was me not `using Distributed` early enough, the code works now! Amazing the only problem I have now is tracking progress through my distributed loop, I can see output from each process in a jobXXXX.out file, but I wrapped my main loop like so:

```julia
if distributed

        println("Distributing work...")
        p = Progress(total_steps)

        progress_pmap(1:total_steps, progress=p) do i

                CODE

```

However, I cannot see this in the error output or in the job output, any ideas?

---

<div class="post-metadata">

**Author:** ![orialb](https://avatars.discourse-cdn.com/v4/letter/o/65b543/32.png) [@orialb](https://discourse.julialang.org/u/orialb)\
**Post date:** [May 7, 2020, 3:19pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/11 "2020-05-07T15:19:50Z")

</div>

Where is `progress_pmap` from? I use ProgressMeter.jl for progress reporting, but I actually only use it when running stuff interactively. Maybe you can try that package?

---

<div class="post-metadata">

**Author:** ![HarrisonWilde](https://avatars.discourse-cdn.com/v4/letter/h/f07891/32.png) [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Post date:** [May 7, 2020, 3:23pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/12 "2020-05-07T15:23:20Z")

</div>

`progress_pmap` is from ProgressMeter.jl as well, I have tried both and neither seems to work. The strange thing is that I think I don’t get any output or error out in the files defined in my sbatch until the programme terminates / crashes, as it is running succesfully right now but none of my printlns in the code during experiment setup have been recorded in those files either.

---

<div class="post-metadata">

**Author:** ![orialb](https://avatars.discourse-cdn.com/v4/letter/o/65b543/32.png) [@orialb](https://discourse.julialang.org/u/orialb)\
**Post date:** [May 7, 2020, 3:29pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/13 "2020-05-07T15:29:23Z")

</div>

Maybe you need to use `flush` somewhere to have the output written to the file? After the job ends you do see all the output from the println statements and so on?

---

<div class="post-metadata">

**Author:** ![HarrisonWilde](https://avatars.discourse-cdn.com/v4/letter/h/f07891/32.png) [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Post date:** [May 7, 2020, 3:32pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/14 "2020-05-07T15:32:39Z")

</div>

Yep I do, so I have a main function that calls stuff like load\_data, sets some constants, splits it up etc. and in between some print lns, so I am wondering if these don’t show up maybe neither will any loading bars?

---

<div class="post-metadata">

**Author:** ![orialb](https://avatars.discourse-cdn.com/v4/letter/o/65b543/32.png) [@orialb](https://discourse.julialang.org/u/orialb)\
**Post date:** [May 7, 2020, 3:46pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/15 "2020-05-07T15:46:03Z")

</div>

Maybe try explicitly printing to a file from within Julia and calling `flush(file_io)`? I think you can also redirect the output of the progress bar when you create it with Progress, but I’m not really sure.

---

<div class="post-metadata">

**Author:** ![HarrisonWilde](https://avatars.discourse-cdn.com/v4/letter/h/f07891/32.png) [@HarrisonWilde](https://discourse.julialang.org/u/HarrisonWilde)\
**Post date:** [May 7, 2020, 3:49pm UTC](https://discourse.julialang.org/t/using-a-machine-file-on-a-cluster-but-also-propagating-environment-to-the-remote-workers/38948/16 "2020-05-07T15:49:36Z")

</div>

Ok will give these things a go, thank you very much for the help!