I am trying to run several jobs on a PBS cluster using ClusterManagers.jl.
using ClusterManagers addprocs_pbs(4,queue="batch")
The job seems to hang:
job id is 24, waiting for job to start .........................................................................................................................................................................................................................
Any advice on a possible fix is welcome!
I ran into this issue last year, as have a few other people (see here and here). It seems like the interactive addprocs_pbs() approach doesn’t work on all clusters. The details of why not I can’t tell you, but an alternative approach is to use Julia’s --machinefile option on startup to tell it which processors PBS has allocated you…basically the cluster version of running julia -p 2 myscript.jl on a laptop:
Thanks! I’ll give it a try.
Related follow-up questions:
My little bit of input here… I think ClusterManagers.jl is constructed to work with OpenPBS.
Your University cluster probably uses PBSPro - which now has a dual open source / commerical support model. Inevitably some of the command line switches will differ.
I did correspond with the author of ClsterManagers.jl and I should have done some testing in a PBSPro environment.
- Is there a way to use this method in an interactive session (IJulia notebook)? It seems to require a command line execution.
Yes, this is something you have to do from the command line. Maybe you could try this workaround, from one of the threads linked above?
- Is it possible to hack the existing ClusterManagers.jl code to accommodate this approach?
I don’t think so, at least not trivially…when you call addprocs_pbs ClusterManagers generates a qsub script and submits it, then finds the addresses of the processors that were started and makes them available to Julia. With the --machinefile approach, you request the processors first, then tell Julia where to find them when it launches.
Though if @johnh is correct, it may just be a matter of sorting out some command-line flags for different flavors of PBS…
In some PBS clusters logfiles for array jobs have a different schematics. Some time ago I sent a patch https://github.com/JuliaParallel/ClusterManagers.jl/issues/85 for getting my universities cluster to work with the package. Seems that the patch is now in the master branch but no yet tagged with a release. So one needs to Pkg.clone("the repo") thing to test that out.
But to see if your PBS cluster is going to work with package you can try to submit a test job from the qsub:
qsub -N test -j oe -k o -t 1:5 ./test
and see if qsub complains. Where the most important part is -t option standing for array jobs.
Perhaps of some interest might be my recent script to start the cluster faster by using TCP/IP to communicate what nodes have been assigned to me for the job:
NP = 200 ### Number of workers expecting
function ip(socket::TCPSocket)
buffer = Array{UInt8}(32)
bufflen::Int64 = 32
ccall(:uv_tcp_getpeername,Int64,(Ptr{Void},Ptr{UInt8},Ptr{Int64}), socket.handle, buffer, &bufflen)
peername::IPv4 = IPv4(buffer[5:8]...)
end
clusterworkers = TCPSocket[]
server = listen(2000)
open(pipeline(`qsub -t 1-$NP`), "w", STDOUT) do io # procs=30
println(io,"nc hpc05 2000")
end
println("Queing...")
for i in 1:NP
conn=accept(server)
push!(clusterworkers,conn)
end
println("Initializing workers...")
cores = [("$(ip(i))",1) for i in clusterworkers]
workerids = addprocs(cores; topology=:master_slave, dir="~",exename="julia")
### Doing some calculation
println(nprocs())
pmap(x->run(`hostname`),1:100)
### Closing the job
rmprocs(workerids)
for i in clusterworkers
close(i)
end
close(server)
@Janis_Erdmanis Am I understanding this correctly?
Your code here submits an array job to PBSPro.
When you then know which compute servers have been alloated to your job you add them as Julia workers using addprocs. Neat!
There is a Python (cough) API available for PBSPro
Similarly there is a Slurm API Slurm Workload Manager - Slurm APIs
I must say that using APIs like that with ClusterManagers.jl would mean coding for each of the cluster managers separately.
However there is an older standard from the Grid Computing days - DRMAA
http://www.drmaa.org/
https://pbspro.atlassian.net/wiki/spaces/PD/pages/49843115/DRMAAV2-3+DRMAAv2+implementation+for+PBSPro
the DRMAA standard should be compatible with all batch systems.
I wonder if we should be getting together a Julia package for DRMAA queries and job submission?
Am I understanding this correctly?
Your code here submits an array job to PBSPro.
When you then know which compute servers have been alloated to your job you add them as Julia workers using addprocs. Neat!
Exactly! The Julia power 
I never had heard about DRMAA. Now I am really curios if I could get jobs submitted with it on my cluster. Sounds like a fun project to get a julia bindings for that.
@Janis_Erdmanis I think DRMAA only makes sense when you have compute jobs being submitted by automated systems, ie some type of biotechnology pipeline or in the case of the Grid by jobs being submitted from other remote systems. Or perhaps if you have a system which submits thousands of jobs, each which has a dependency on a job which went before it.
In fact the last company I worked with had a job system just like that. Did not use DRMAA though!
I agree that a DRMAA port for Julia would be interesting.
Hi guys, sorry but is there any update in this?
Can tell PBS to use ncpus=64, then use Distributed.jl’s addprocs instead? I am currently using this approach and it seems alright, i.e. I get logging from each worker.
From my understanding is ncpus requests to get CPUs on a node and thus, Distributed.addprocs seems fine in this case?
Thanks.