# Debugging possible issue with \`machinefile\` option on SLURM system
**URL:** https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857
**Category:** Julia at Scale
**Created:** [December 19, 2017, 5:49pm UTC](https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857 "2017-12-19T17:49:50Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@nickeubank](https://discourse.julialang.org/u/nickeubank)
#### Post date: [December 19, 2017, 5:49pm UTC](https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857/1 "2017-12-19T17:49:50Z")
Trying to debug a problem that may be due to `julia` not using a provided `nodefile` on a SLURM system. It appears that jobs are getting spun-off on hosts where it shouldn’t be.
The SLURM batch script is:
```julia
#!/bin/bash
#SBATCH --mail-user=myemail@email.com
#SBATCH --mail-type=ALL # Alerts sent when job begins, ends, or aborts
#SBATCH --ntasks=16
#SBATCH --ntasks-per-node=8
#SBATCH --mem=100G
#SBATCH --job-name=indiv_array
#SBATCH --array=1-5
#SBATCH --time=03-00:00:00 # Wall Clock time (dd-hh:mm:ss) [max of 14 days]
#SBATCH --output=indiv_array_%A_%a.output # output and error messages go to this file
export SLURM_NODEFILE=`generate_pbs_nodefile`
julia --machinefile $SLURM_NODEFILE indiv_array.jl
```
Does anyone (a) see anything wrong with this, and (b) have any suggestions how I can check (from within the resulting Juila session) which nodes are actually being used to see if they correspond to the ones generated in `SLURM_NODEFILE`? i.e. is there a function to report where `julia` thinks it’s supposed to be running tasks I can print out?
---
### Author:  [@raminammour](https://discourse.julialang.org/u/raminammour)
#### Post date: [December 19, 2017, 6:16pm UTC](https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857/2 "2017-12-19T18:16:53Z")
You could use `gethostname()` or even `run(`hostname`)`. However, my recommendation would be to take a look at:
[https://github.com/JuliaParallel/ClusterManagers.jl](https://github.com/JuliaParallel/ClusterManagers.jl)
And use the `slurm` manager.
---
### Author:  [@nickeubank](https://discourse.julialang.org/u/nickeubank)
#### Post date: [December 19, 2017, 6:30pm UTC](https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857/3 "2017-12-19T18:30:33Z")
Thanks @raminammour! `gethostname()` is perfect.
Re: cluster managers: is there something wrong with `machinefile`? I can’t find any notes about it being depreciated or anything. Seems like it should be pretty straightforward, aside from the fact it isn’t working. 😕
I’ve looked into the slurm manager, but I don’t think it’ll work on our system – sounds like I need a running Julia instance to manage all the spin-off procs created by the `srun` execution. The problem is I can only run code on a gateway system, which wouldn’t leave that original julia instance running long enough to manage those spinoff processes.
---
### Author:  [@raminammour](https://discourse.julialang.org/u/raminammour)
#### Post date: [December 19, 2017, 6:37pm UTC](https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857/4 "2017-12-19T18:37:16Z")
What has worked on my system, is requesting one node in interactive mode and using it to spawn all the other processes, or maybe even in batch mode.
I have used `machinefile` on an `lsf` system a long time ago, it used to work. However, keep in mind that you would be using SSH to connect to all of them, you may run out of ports if you have a big enough job and it is slower anyway.
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [December 19, 2017, 6:39pm UTC](https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857/5 "2017-12-19T18:39:01Z")
> [@nickeubank](#):
>
> Re: cluster managers: is there something wrong with machinefile? I can’t find any notes about it being depreciated or anything. Seems like it should be pretty straightforward, aside from the fact it isn’t working.
You’ll need to find out where it is on your system. Are you sure it’s located at `generate_pbs_nodefile`? What directory are you running this from?
If you startup an MPI job you can just look for where the file pops up and save that location. It can be different on each cluster.
---
### Author:  [@nickeubank](https://discourse.julialang.org/u/nickeubank)
#### Post date: [December 19, 2017, 6:41pm UTC](https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857/6 "2017-12-19T18:41:38Z")
> [@raminammour](#):
>
> What has worked on my system, is requesting one node in interactive mode and using it to spawn all the other processes, or maybe even in batch mode.
I appreciate that, but I think that’s gonna get real complicated in our system (it’s a university system with long waits for access). Thus the use of `sbatch`. I’d have to (a) ask for an interactive session that would last longer than my task, then (b) when it comes up, request the resources for the real jobs and hope they become available in time.
> [@ChrisRackauckas](#):
>
> You’ll need to find out where it is on your system. Are you sure it’s located at generate\_pbs\_nodefile? What directory are you running this from?
Hmmm… I was told by the IT folks that that would execute and give the right path, but maybe they were sloppy when they gave me that and make implicit assumptions about where I was working from. I’ll investigate that.
---
### Author:  [@nickeubank](https://discourse.julialang.org/u/nickeubank)
#### Post date: [December 19, 2017, 7:00pm UTC](https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857/7 "2017-12-19T19:00:34Z")
@ChrisRackauckas
So `generate_pbs_nodefile` returns an absolute path (e.g. `/tmp/oQ8vxZUMRR`) and returns a text file like this:
```julia
[eubankn@vm-qa-node001 slurm]$ cat /tmp/oQ8vxZUMRR
vm-qa-node001
```
Seem appropriate?
---
### Author:  [@nickeubank](https://discourse.julialang.org/u/nickeubank)
#### Post date: [December 19, 2017, 7:04pm UTC](https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857/8 "2017-12-19T19:04:14Z")
@ChrisRackauckas Here’s a two-tasks on two-nodes version:
```julia
[eubankn@vm-qa-node001 slurm]$ salloc --partition=debug --nodes=2 --ntasks=2 --tasks-per-node=1
salloc: Granted job allocation 22594479
[eubankn@vm-qa-node001 slurm]$ generate_pbs_nodefile
/tmp/UUMuRd3rwj
[eubankn@vm-qa-node001 slurm]$ cat `generate_pbs_nodefile`
vm-qa-node001
vm-qa-node002
```
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [December 19, 2017, 7:08pm UTC](https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857/9 "2017-12-19T19:08:43Z")
Yeah, that looks appropriate if the compute nodes you’re supposed to be using are named `vm-qa-node001` and `vm-qa-node002`.
---
### Author:  [@nickeubank](https://discourse.julialang.org/u/nickeubank)
#### Post date: [December 19, 2017, 7:10pm UTC](https://discourse.julialang.org/t/debugging-possible-issue-with-machinefile-option-on-slurm-system/7857/10 "2017-12-19T19:10:32Z")
yeah… 😕 OK, well, thanks for confirming I’m not doing something super stupid! Have reservation at 5pm to actually check whether parallel jobs report those as their hostnames. Fingers crossed!