# ClusterManagers.jl hangs on pbs

**URL:** <https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056>\
**Category:** Julia at Scale\
**Tags:** parallel\
**Created:** [May 21, 2018, 6:34pm UTC](https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056 "2018-05-21T18:34:13Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Snir\_Gazit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/snir_gazit/32/11027_2.png) [@Snir\_Gazit](https://discourse.julialang.org/u/Snir_Gazit)\
**Post date:** [May 21, 2018, 6:34pm UTC](https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056/1 "2018-05-21T18:34:13Z")

</div>

I am trying to run several jobs on a PBS cluster using ClusterManagers.jl.

`using ClusterManagers addprocs_pbs(4,queue="batch") `

The job seems to hang:

```julia
job id is 24, waiting for job to start .........................................................................................................................................................................................................................

```

Any advice on a possible fix is welcome!

---

<div class="post-metadata">

**Author:** ![ElOceanografo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eloceanografo/32/624_2.png) [@ElOceanografo](https://discourse.julialang.org/u/ElOceanografo)\
**Post date:** [May 22, 2018, 12:05am UTC](https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056/2 "2018-05-22T00:05:14Z")

</div>

I ran into this issue last year, as have a few other people (see [here](https://discourse.julialang.org/t/help-setting-up-julia-on-a-cluster/5519) and [here](https://discourse.julialang.org/t/clustermanagers-package-not-connecting-for-pbs-batch-system/8879/5)). It seems like the interactive `addprocs_pbs()` approach doesn’t work on all clusters. The details of why not I can’t tell you, but an alternative approach is to use Julia’s `--machinefile` option on startup to tell it which processors PBS has allocated you…basically the cluster version of running `julia -p 2 myscript.jl` on a laptop:

> [@Help setting up Julia on a cluster](https://discourse.julialang.org/t/help-setting-up-julia-on-a-cluster/5519/6):
>
> I recently went through the same learning process on my university’s cluster, and your second question was the hardest part for me. I found the easiest way was to submit to one of the queues using a PBS submit script that tells Julia where to find all the processors via the PBS nodefile. Here’s a minimum working example PBS script: #!/bin/bash #PBS -l nodes=4:ppn=12,walltime=00:05:00 #PBS -N test\_julia #PBS -q debug echo PBS: node file is $PBS\_NODEFILE julia --machinefile=$PBS\_NODEFILE /path…

---

<div class="post-metadata">

**Author:** ![Snir\_Gazit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/snir_gazit/32/11027_2.png) [@Snir\_Gazit](https://discourse.julialang.org/u/Snir_Gazit)\
**Post date:** [May 22, 2018, 5:59am UTC](https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056/3 "2018-05-22T05:59:44Z")

</div>

Thanks! I’ll give it a try.

Related follow-up questions:

1. Is there a way to use this method in an interactive session (IJulia notebook)? It seems to require a command line execution.
2. Is it possible to hack the existing ClusterManagers.jl code to accommodate this approach?

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [May 22, 2018, 11:37am UTC](https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056/4 "2018-05-22T11:37:41Z")

</div>

My little bit of input here… I think ClusterManagers.jl is constructed to work with OpenPBS.  
Your University cluster probably uses PBSPro - which now has a dual open source / commerical support model. Inevitably some of the command line switches will differ.  
I did correspond with the author of ClsterManagers.jl and I should have done some testing in a PBSPro environment.

---

<div class="post-metadata">

**Author:** ![ElOceanografo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eloceanografo/32/624_2.png) [@ElOceanografo](https://discourse.julialang.org/u/ElOceanografo)\
**Post date:** [May 22, 2018, 4:54pm UTC](https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056/5 "2018-05-22T16:54:34Z")

</div>

> 1. Is there a way to use this method in an interactive session (IJulia notebook)? It seems to require a command line execution.

Yes, this is something you have to do from the command line. Maybe you could try [this workaround](https://discourse.julialang.org/t/clustermanagers-package-not-connecting-for-pbs-batch-system/8879/8), from one of the threads linked above?

> 1. Is it possible to hack the existing ClusterManagers.jl code to accommodate this approach?

I don’t think so, at least not trivially…when you call `addprocs_pbs` ClusterManagers [generates a qsub script](https://github.com/JuliaParallel/ClusterManagers.jl/blob/63770d62e3b19572101b94f18e3b2c4297aeebdc/src/qsub.jl#L71) and submits it, then finds the addresses of the processors that were started and makes them available to Julia. With the `--machinefile` approach, you request the processors first, then tell Julia where to find them when it launches.

Though if @johnh is correct, it may just be a matter of sorting out some command-line flags for different flavors of PBS…

---

<div class="post-metadata">

**Author:** ![Janis\_Erdmanis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janis_erdmanis/32/10869_2.png) [@Janis\_Erdmanis](https://discourse.julialang.org/u/Janis_Erdmanis)\
**Post date:** [May 22, 2018, 6:34pm UTC](https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056/6 "2018-05-22T18:34:53Z")

</div>

In some PBS clusters logfiles for array jobs have a different schematics. Some time ago I sent a patch [https://github.com/JuliaParallel/ClusterManagers.jl/issues/85](https://github.com/JuliaParallel/ClusterManagers.jl/issues/85) for getting my universities cluster to work with the package. Seems that the patch is now in the master branch but no yet tagged with a release. So one needs to `Pkg.clone("the repo")` thing to test that out.

But to see if your PBS cluster is going to work with package you can try to submit a test job from the `qsub`:

```julia
qsub -N test -j oe -k o -t 1:5 ./test

```

and see if `qsub` complains. Where the most important part is `-t` option standing for array jobs.

Perhaps of some interest might be my recent script to start the cluster faster by using TCP/IP to communicate what nodes have been assigned to me for the job:

```julia
NP = 200 ### Number of workers expecting

function ip(socket::TCPSocket)
    buffer = Array{UInt8}(32)
    bufflen::Int64 = 32
    ccall(:uv_tcp_getpeername,Int64,(Ptr{Void},Ptr{UInt8},Ptr{Int64}), socket.handle, buffer, &bufflen)
    peername::IPv4 = IPv4(buffer[5:8]...)
end

clusterworkers = TCPSocket[]

server = listen(2000)

open(pipeline(`qsub -t 1-$NP`), "w", STDOUT) do io # procs=30
    println(io,"nc hpc05 2000")
end

println("Queing...")
for i in 1:NP
    conn=accept(server)
    push!(clusterworkers,conn)
end

println("Initializing workers...")
cores = [("$(ip(i))",1) for i in clusterworkers]
workerids = addprocs(cores; topology=:master_slave, dir="~",exename="julia")

### Doing some calculation

println(nprocs())
pmap(x->run(`hostname`),1:100)

### Closing the job

rmprocs(workerids)
for i in clusterworkers
    close(i)
end
close(server)

```

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [May 24, 2018, 9:03am UTC](https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056/7 "2018-05-24T09:03:58Z")

</div>

@Janis_Erdmanis Am I understanding this correctly?  
Your code here submits an array job to PBSPro.  
When you then know which compute servers have been alloated to your job you add them as Julia workers using addprocs. Neat!

There is a Python (cough) API available for PBSPro  
Similarly there is a Slurm API [Slurm Workload Manager - Slurm APIs](https://slurm.schedmd.com/api.html)  
I must say that using APIs like that with ClusterManagers.jl would mean coding for each of the cluster managers separately.

However there is an older standard from the Grid Computing days - DRMAA  
[http://www.drmaa.org/](http://www.drmaa.org/)  
[https://pbspro.atlassian.net/wiki/spaces/PD/pages/49843115/DRMAAV2-3+DRMAAv2+implementation+for+PBSPro](https://pbspro.atlassian.net/wiki/spaces/PD/pages/49843115/DRMAAV2-3+DRMAAv2+implementation+for+PBSPro)

> **[GFD.231.pdf](https://www.ogf.org/documents/GFD.231.pdf)**
>
> 395.33 KB

the DRMAA standard should be compatible with all batch systems.  
I wonder if we should be getting together a Julia package for DRMAA queries and job submission?

---

<div class="post-metadata">

**Author:** ![Janis\_Erdmanis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janis_erdmanis/32/10869_2.png) [@Janis\_Erdmanis](https://discourse.julialang.org/u/Janis_Erdmanis)\
**Post date:** [May 24, 2018, 1:33pm UTC](https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056/8 "2018-05-24T13:33:32Z")

</div>

> Am I understanding this correctly?  
> Your code here submits an array job to PBSPro.  
> When you then know which compute servers have been alloated to your job you add them as Julia workers using addprocs. Neat!

Exactly! The Julia power 🙂

I never had heard about DRMAA. Now I am really curios if I could get jobs submitted with it on my cluster. Sounds like a fun project to get a julia bindings for that.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [May 24, 2018, 1:46pm UTC](https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056/9 "2018-05-24T13:46:24Z")

</div>

@Janis_Erdmanis I think DRMAA only makes sense when you have compute jobs being submitted by automated systems, ie some type of biotechnology pipeline or in the case of the Grid by jobs being submitted from other remote systems. Or perhaps if you have a system which submits thousands of jobs, each which has a dependency on a job which went before it.  
In fact the last company I worked with had a job system just like that. Did not use DRMAA though!

I agree that a DRMAA port for Julia would be interesting.

---

<div class="post-metadata">

**Author:** ![darrencl](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/darrencl/32/8552_2.png) [@darrencl](https://discourse.julialang.org/u/darrencl)\
**Post date:** [March 3, 2020, 11:37pm UTC](https://discourse.julialang.org/t/clustermanagers-jl-hangs-on-pbs/11056/10 "2020-03-03T23:37:57Z")

</div>

Hi guys, sorry but is there any update in this?

Can tell PBS to use `ncpus=64`, then use `Distributed.jl`’s `addprocs` instead? I am currently using this approach and it seems alright, i.e. I get logging from each worker.

From my understanding is `ncpus` requests to get CPUs on a node and thus, `Distributed.addprocs` seems fine in this case?

Thanks.