# Multinode scaling

**URL:** https://discourse.julialang.org/t/multinode-scaling/70503
**Category:** Julia at Scale
**Tags:** hpc, distributed
**Created:** [October 27, 2021, 7:30pm UTC](https://discourse.julialang.org/t/multinode-scaling/70503 "2021-10-27T19:30:48Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![Antonis\_Skourtis](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@Antonis\_Skourtis](https://discourse.julialang.org/u/Antonis_Skourtis)
#### Post date: [October 27, 2021, 7:30pm UTC](https://discourse.julialang.org/t/multinode-scaling/70503/1 "2021-10-27T19:30:48Z")

</div>

Hello,

While I was experimenting with the `Distributed` library I ran into some issue. I am trying to utilize multiple nodes on a given cluster using Slurm but as of yet I cannot get it working.

My `script.jl` is quite simple

```julia
using Distributed

# Define what id() is
@everywhere id() = (myid(), gethostname())

# Run id() on all nodes
ids = [id(), [@fetchfrom i id() for i in workers()]...]

# Print
println.(ids)

#EOF

```

And I’m submitting my job with the given `submit.sh` script

```bash
#!/bin/bash
#SBATCH --job-name=julia-demo
#SBATCH --time=00:01:00
#SBATCH --nodes=2
#SBATCH --partition=testing
#SBATCH --output=log.out
#SBATCH --error=log.err

module load gcc/10.2.0 julia
export NODEFILE=`generate_pbs_nodefile`
srun julia --machine-file $NODEFILE ./script.jl 

```

This didn’t work as expected and so I thought that I probably need to run the script from my entry node and julia will spawn my workers on the allocated computing nodes of my cluster.

So I tried this approach instead `$ salloc --nodes=2 --partition=testing julia --machine-file machinefile script.jl`

`machinefile` is a file containing the hostnames of the allocated nodes

This approach yielded the following error

```julia
Host key verification failed.
Permission denied, please try again.
Permission denied, please try again.
Permission denied (gssapi-keyex,gssapi-with-mic,password).
ERROR: TaskFailedException

    nested task error: Unable to read host:port string from worker. Launch command exited with error?
    Stacktrace:
    ...

```

Please give me a simple example to get started with multinode scaling, without the use of `ClusterManagers`

Thank you for your time and help!

---

<div class="post-metadata">

### Author: ![jishnub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jishnub/32/33620_2.png) [@jishnub](https://discourse.julialang.org/u/jishnub)
#### Post date: [October 28, 2021, 7:03am UTC](https://discourse.julialang.org/t/multinode-scaling/70503/2 "2021-10-28T07:03:56Z")

</div>

Try something like this:

jobscript (modify as necessary):

```bash
#!/bin/bash
#SBATCH --job-name=julia-demo
#SBATCH --time=00:01:00
#SBATCH --nodes=2
#SBATCH --output=log.out
#SBATCH --error=log.err

cd $SCRATCH/temp
julia=$SCRATCH/julia/julia-1.7.0-rc2/bin/julia
$julia script.jl

```

script.jl

```julia
using ClusterManagers
# the jobscript sets the environment variable SLURM_NNODES
# we read it in here to launch the appropriate number of jobs
# In general we need to know which variable is set in the jobscript
N = parse(Int, ENV["SLURM_NNODES"])
addprocs_slurm(N, N = N)
using Distributed
@show workers()

# Define what id() is
@everywhere id() = (myid(), gethostname())

# Run id() on all nodes
ids = [id(), [@fetchfrom i id() for i in workers()]...]

# Print
println.(ids)

rmprocs.(workers())

```

output

```julia
temp/ $ cat log.err
temp/ $ cat log.out
connecting to worker 1 out of 2
connecting to worker 2 out of 2
workers() = [2, 3]
(1, "compute-19-12.local")
(2, "compute-19-12.local")
(3, "compute-24-18.local")

```

---

<div class="post-metadata">

### Author: ![Antonis\_Skourtis](https://avatars.discourse-cdn.com/v4/letter/a/ed655f/32.png) [@Antonis\_Skourtis](https://discourse.julialang.org/u/Antonis_Skourtis)
#### Post date: [October 28, 2021, 8:25am UTC](https://discourse.julialang.org/t/multinode-scaling/70503/3 "2021-10-28T08:25:50Z")

</div>

Thank you for your response but like I mentioned in my original question, I don’t want to use cluster managers, or any other library for that matter. Since this is a learning experience for me I would like to do it as bare bones as possible.