I have access to a cluster with multiple nodes, connected thorugh SSH, each of which has multiple CPUs. My algorithm uses DistributedArrays.jl to create a distributed array and have multiple processes work on it: at the end of the simulation, the result is returned to the master worker. Since I have…

I may have found a solution. I’ve noticed that, each time I add nodes using addprocs(p) where p is something like p = [("node1", 3), ("node2", 3), ("node3", 3), ...], the first N workers of the vector returned by workers() each belong to one of the N added nodes. For example, if I add 3 nodes, what…

Did you get this issue resolved for good?

Your solution with WorkPool for sure is the easiest to implement. Another approaches would be: create an algorithm that sends parameters to specific nodes with @spawnat create small remote services waiting to consume data from a RemoteChannel, and you put your input values inside the right channel …

How to have each node in a cluster run a parallel job on its own CPUs, using SSH?

Specific Domains Julia at Scale

draftman9 October 14, 2024, 11:05am 5

Maybe like this: RemoteChannels and workload queue architecture - #2 by draftman9

Topic		Replies	Views
Running a function over multiple nodes on a julia cluster, from within a julia script Julia at Scale hpc , cluster , distributed , slurm	5	299	August 14, 2024
Running a process on several nodes on cluster Performance cluster , distributed	12	1674	December 6, 2022
Getting started with distributed Julia computations on a cluster Julia at Scale	1	615	September 27, 2020
Correct way of parallelizing on a HPC remote cluster machine Performance question , hpc , parallel , distributed , threads	8	1399	August 25, 2020
Extend distributed pmap to mutliple nodes using SLURM Julia at Scale hpc , parallel , distributed , pmap , slurm	2	348	May 3, 2023

How to have each node in a cluster run a parallel job on its own CPUs, using SSH?

Related topics