Julia on Cluster with SSH Restriction

question
cluster

#1

I’m attempting to use Julia on a cluster (using PBS) that blocks ssh connections to anything except the head node. Based on the documentation, it seems that Julia requires passwordless ssh to start workers on cluster nodes:

The base Julia installation has in-built support for two types of clusters:

  • A local cluster specified with the -p option as shown above.
  • A cluster spanning machines using the --machinefile option. This uses a passwordless ssh login to start Julia worker processes (from the same path as the current host) on the specified machines.

I’ve tried the solutions presented on this thread, but the following errors occur: (1) ClusterManagers hangs when calling addprocs_pbs() or (2) I get a permissions error when the ssh connection is attempted.

Permission denied, please try again.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
ERROR: Unable to read host:port string from worker. Launch command exited with error?
read_worker_host_port(::Pipe) at ./distributed/cluster.jl:236
connect(::Base.Distributed.SSHManager, ::Int64, ::WorkerConfig) at ./distributed/managers.jl:391
create_worker(::Base.Distributed.SSHManager, ::WorkerConfig) at ./distributed/cluster.jl:443
setup_launched_worker(::Base.Distributed.SSHManager, ::WorkerConfig, ::Array{Int64,1}) at ./distributed/cluster.jl:389
(::Base.Distributed.##33#36{Base.Distributed.SSHManager,WorkerConfig,Array{Int64,1}})() at ./task.jl:335

My SysAdmin seems unwilling to allow ssh connections to worker nodes. Is there another option for using Julia on the cluster that bypasses this problem?


#2

@Brosetti I have installed and managed PBS clusters. I think what is happening is that the PAM module for PBS is installed. Yes, this stops a user from sshing into the compute nodes. BUT when you are running a job you should be able to ssh into the compute nodes which are allocated to you.
You will find that there is an environment variable called PBS_NODELIST which gives you the list of compute nodes you can use.

Please give me five minutes and I will confirm the PAM module behaviour.

It could be that you have some other type of restriction though?

There is a very old style pbs_dsh utility whihc might be put into a wrapper script and substituted for ssh. But lets not go there.


#3

What I said is true for Torque
http://docs.adaptivecomputing.com/torque/3-0-5/3.4hostsecurity.php

My Google-fu is exhausted in looking for the PBSPro style.
May I suggest that you start an interctive job using qsub -i Once you have started a shel lon the first compute node see if you can ssh into the others?
I guess not as you have already shown us this output…


#4

Another option you may have is the MPI ClusterManager, see e.g. here:

This is the currently released version, there is also my version that uses one-sided MPI calls here, but it’s not merged to MPI.jl yet:

Either of these options should give you native Julia parallel calls using MPI as the communication layer, bypassing the need for ssh entirely. I have only tested this with some basic DistributedArrays stuff, so I’m not sure how well it holds up.


#5

I would also like to ask. How are parallel jobs run on your cluster? I would guess by using an MPI with ‘munge’ authentication.
https://dun.github.io/munge/
Which leads me to open the discussin with the Julia developers. Maybe a plugin style is needed for Julia parallel, and do not assume that only ssh will be used.

Its worth saying that the original PBS was invented well before ssh was around and it used the r protocols - rsh rcp rlogin They have the wonderfulness of the original restriction to six character user names.
In addition to relying on dotfiles for security!


#6

Hi @barche But surely MPI needs to be launched somehow… and there are a variety of ways in which that is done. One of them being ssh!
I may well be barking up the wrong branch here (Git pun intended).


#7

Yep, with any luck by simply doing mpirun julia mysimulation.jl

I tested this with slurm, there mpirun gets the nodes to run on and so forth directly from the task manager.


#8

@barche Slurm should be using munge for the mpirun startup phase
And yes slurm is pretty cute - it has good integration.
I will confess that the first time I met a slurm cluster I did the normal job submisssion script / find the list of nodes / create a custom hostfile. Till someone pointed out all you have to do is the srun… doooh


#9

Thanks for the quick replies fellas. Let’s see if I can respond to your comments/questions:

Yes. I currently use this file in my pbs script. Here is what produced the errors that I reported above:

myjob.pbs

#!/bin/bash
#PBS -l nodes=12:ppn=4,walltime=00:05:00
#PBS -N test
#PBS -q batch

julia --machinefile $PBS_NODEFILE test.jl

test.jl

println("Hello from Julia")
addprocs(48)
np = nprocs()
println("Number of processes: $np")


@sync @parallel for i=1:48
  host = gethostname()
  pid = getpid()
  sleep(2)
  println("I am $host - $pid doing loop $i")
end

PAM is an interesting feature, but I don’t think this is what my SysAdmin is using. I believe he has globally blocked all users but root from ssh-ing to anything but the head node at all times. I’ll suggest that he look into PAM as an alternative.


The process hangs when I try interactive mode using qsub -I. It never gets past qsub: waiting for job to start. I suspect this has something to do with the global ssh block.


Thanks for this suggestion! I’m getting some build errors related to MPI_C. Once I sort them out, I’ll reply back with my results


This is my first time using the cluster, so I’ll have to look into the details.


#11

I suggest the following. HPC Admins portray themselves as Ogres. Heavy metal T-shirt? Combat boots? Black jeans?
Bring a packet of cookies. Even better some local craft beer.

Actually, ssh restriction like this are put there to stop users doing stupid things. The admin will want you to use his/her system (*). Some explanation of how Julia works and how great it is may wake the Ogre.

(*) Women can wear combat boots and T-shirts. I refer you to Lady Fiona in Shrek.


#12

OK, feel free to post the error here or in an MPI.jl issue if you get stuck on it. Looking at your script, it should be sufficient to replace the current julia command with mpirun julia test.jl (assuming mpirun supports PBS, which it should), and enclose your test script in:

using MPI

mgr = MPI.start_main_loop(MPI.MPI_TRANSPORT_ALL)

# code

MPI.stop_main_loop(mgr)

#13

Try doing:

export CC=mpicc
export FC=mpif90
export CXX=mpicxx

in the shell before building MPI.jl. How well CMake can find MPI installation seems to vary widely with CMake version, and setting these variables helps it.