Distributed.jl: Group myid() by computing nodes?

[lmtzx9h4qqnt]

Using Distributed.jl, is there no built-in way to find out which workers are on the same shared-memory node? This post:

@spawnat with SlurmClusterManager

It seems the following works, though isn’t amazingly pretty. Thoughts appreciated. @everywhere begin function return_id(x) return myid(), gethostname() end end nworkers = length(workers()) out = pmap(return_id,1:nworkers) idlst = map(x->x[1],out) hostlist = map(x->x[2],out) df = DataFrame(id=idlst,host=hostlist) gp = groupby(df, :host) gvec = gp.groups nodeworkers = [ ] for i = 1:maximum(gvec) push!(nodeworkers,df.id[gvec.==i]) end Then you can manually build the SharedArra…

shows a manual implementation based on gethostname(), but I don’t know how robust that is? If there’s nothing built-in, maybe there’s already a package that helps with this?

My goal is to create one SharedArray per cluster node, which is accessible to all workers on that node, so I need to find out which machine each of the workers is running on.

[johnh]

Is this any help?

How to get more info on w ∈ workers()

1 i’m not sure about that but seems RemoteException type is for this purpose, maybe you can refer to julia manual here Distributed Computing · The Julia Language 2 you can use Distributed.map_pid_wrkr function to map the pid to a Worker type, then you can lookup the hostname, such as Distributed.map_pid_wrkr[pid].config.host. config field is a WorkConfig type, and all field name can be obtained by using Distrbuted; fieldnames(WorkerConfig), that is (:io, :host, :port, :count, :exename, :exeflag…

[lmtzx9h4qqnt]

Well, since Distributed.map_pid_wrkr is not documented or exported, I’d be hesitant to rely on it. And it seems like the suggestion pretty much the equivalent of pmap(_ -> gethostname(), procs())?

[lmtzx9h4qqnt]

I just noticed that Distributed does at least export check_same_host(pids) (no documentation, though), which also seems to be used by SharedArrays, and while that does allow one to check if the given processes are on the same node, it’d be pretty cumbersome and inefficient to use that to partition the processes by their nodes.