# Is ClusterManagers.jl maintained? Or, how to do multi-node calculations in Julia?
**URL:** https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050
**Category:** General Usage
**Tags:** question, package
**Created:** [February 10, 2024, 11:27pm UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050 "2024-02-10T23:27:33Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)
#### Post date: [February 10, 2024, 11:27pm UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/1 "2024-02-10T23:27:33Z")
Just wondering if anybody is maintaining [ClusterManagers.jl](https://github.com/JuliaParallel/ClusterManagers.jl)? Major components of it have been broken for over two years now… ([example](https://github.com/JuliaParallel/ClusterManagers.jl/issues/179)). Now it seems that even the previously well-supported cluster managers are a bit sketchy ([Slurm broken · Issue #196 · JuliaParallel/ClusterManagers.jl · GitHub](https://github.com/JuliaParallel/ClusterManagers.jl/issues/196)).
Distributed compute is such an important part of a scientific computing, I’m a bit worried that this part of the ecosystem is in such a dismal state.
If ClusterManagers is not a reliable option, how do people typically do distributed computation in Julia? Maybe there’s some package everybody is using these days that I am not aware of…
---
### Author:  [@adienes](https://discourse.julialang.org/u/adienes)
#### Post date: [February 10, 2024, 11:44pm UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/2 "2024-02-10T23:44:30Z")
I tried to find a solution for this last year and gave up after a while and ended up using [ray](https://github.com/ray-project/ray), which has gone pretty smoothly
I suspect you’ll have an easier time using python/ray to do all the cluster stuff and using `subprocess.run` to launch Julia processes, than you will trying to do everything in Julia
---
### Author:  [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)
#### Post date: [February 10, 2024, 11:58pm UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/3 "2024-02-10T23:58:27Z")
Thanks. Do you call Ray from Julia(/PythonCall)? Or do you mean you launch your entire workflow from Python?
Also, how do you pass Julia objects between workers? (Maybe just serializing/unserializing to a file on a shared file system?)
---
### Author:  [@adienes](https://discourse.julialang.org/u/adienes)
#### Post date: [February 11, 2024, 12:11am UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/4 "2024-02-11T00:11:09Z")
when in a cluster/distributed context, I try to stick to the basics and avoid any workflow that sounds even mildly complex 😅
I launch everything from python, but use `subprocess.run` to call the workhorse (julia or c++). probably `PythonCall` would work but I haven’t tried it.
if I need an object back to pass back and forth, I’ll try to serialize to a file inside the subprocess (julia), then read it back when the subprocess returns to the ray (python) worker which can put it into the distributed object store. so the serialization format has to be something both languages can understand. in my case that’s parquet but there are probably better options
note my workflow is “embarrassingly parallel” and mostly linear, so ymmv
---
### Author:  [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)
#### Post date: [February 11, 2024, 12:17am UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/5 "2024-02-11T00:17:42Z")
Thanks, I see. Yeah I definitely need to have fairly frequent communication all-to-all between workers so sadly this wouldn’t work for me. But I will still check out ray for launching… maybe there’s a super simple way to just have every worker communicate via files. (My stuff is not high throughput, it just needs to be fairly regular and asynchronous)
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [February 11, 2024, 1:17am UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/6 "2024-02-11T01:17:48Z")
> [@MilesCranmer](#):
>
> how do people typically do distributed computation in Julia?
I think MPI. But our infrastructure don’t have MPI, so most of the time the answer is we don’t…
---
### Author:  [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)
#### Post date: [February 11, 2024, 2:31am UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/7 "2024-02-11T02:31:44Z")
Would Dagger.jl work for you?
> **[GitHub - JuliaParallel/Dagger.jl: A framework for out-of-core and parallel...](https://github.com/JuliaParallel/Dagger.jl)**
>
> A framework for out-of-core and parallel execution - GitHub - JuliaParallel/Dagger.jl: A framework for out-of-core and parallel execution
---
### Author:  [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)
#### Post date: [February 11, 2024, 2:53am UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/8 "2024-02-11T02:53:04Z")
Dagger.jl is a high-level wrapper of Distributed.jl, so doesn’t by itself handle any multi-node stuff (until someone fixes ClusterManagers.jl)
---
### Author:  [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)
#### Post date: [February 11, 2024, 3:04am UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/9 "2024-02-11T03:04:32Z")
Thanks; do you mean people manually make calls via MPI.jl? I checked out MPIClusterManagers.jl but it also seems pretty broken…
It’s really weird to me that the distributed computing side of Julia is this deficient. This seems like one of the main things Julia would be really good at given the community’s focus on scientific computing… Am I missing something here? Is everybody is just batch submitting single-node Julia scripts and we among a small handful of people trying to do true multi-node stuff?
---
### Author:  [@martinmestre](https://discourse.julialang.org/u/martinmestre)
#### Post date: [February 11, 2024, 3:29am UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/10 "2024-02-11T03:29:24Z")
Hi Miles, I have used SlurmClusterManager.jl for 3 nodes.
> **[GitHub - kleinhenz/SlurmClusterManager.jl: julia package for running code on...](https://github.com/kleinhenz/SlurmClusterManager.jl)**
>
> julia package for running code on slurm clusters. Contribute to kleinhenz/SlurmClusterManager.jl development by creating an account on GitHub.
But my code is not complex in parallel terms. Please post here if you find a solution.
Best
---
### Author:  [@adienes](https://discourse.julialang.org/u/adienes)
#### Post date: [February 11, 2024, 4:06am UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/11 "2024-02-11T04:06:07Z")
there is also `https://github.com/ray-project/ray/pull/40098` and `https://github.com/beacon-biosignals/Ray.jl` which look exciting, though I have used neither
---
### Author:  [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)
#### Post date: [February 11, 2024, 6:48am UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/12 "2024-02-11T06:48:21Z")
People that really care about distributed computing, especially involving more than just two or three nodes, use MPI. It’s the industry standard and well supported in Julia through MPI.jl. The (big) drawback is that you pretty much loose all interactivity.
As I would describe it, Distributed.jl is the attempt to enable the user to do convenient and interactive small scale distributed computing. It is a non standard approach in a less explored design space. People that have been creating and working on it have focused on pushing Julia on other fronts (e.g @vchuravy has brought us pkgimages). And attracting new people to work on it has been difficult because, among other things, it has lived in the main Julia repo. Hopefully we will see more activity when it’s separate package.
---
### Author:  [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)
#### Post date: [February 11, 2024, 2:41pm UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/13 "2024-02-11T14:41:52Z")
Thanks, this is very helpful! I will switch to MPI.jl then.
I think my mistake was assuming Distributed.jl being in the stdlib meant it was the standard Julia approach for anything involving distributed compute. (Whereas MPI.jl being a package meant it was an experiment.)
---
### Author:  [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)
#### Post date: [February 11, 2024, 7:02pm UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/14 "2024-02-11T19:02:48Z")
Argh, I guess this means you couldn’t do distributed compute from PythonCall.jl then, right? Since you have to launch Julia via mpiexecjl? Looks like I don’t really have any good options here… All I really want is to start up Julia and execute an operation over my slurm allocation. One year ago this seemed to work with ClusterManagers but now things seem to have broken ☹
---
### Author:  [@vchuravy](https://discourse.julialang.org/u/vchuravy)
#### Post date: [February 11, 2024, 7:18pm UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/15 "2024-02-11T19:18:18Z")
ClusterManagers.jl is distinct from Distributed.jl – the latter being a standard library and quite stable (it could benefit from a dedicated maintainer, but it’s stability is also a boon).
ClusterManagers.jl is an external package that provides integration with diverse schedulers and has shown itself to be challenging to maintain partly because of the diversity of schedulers and the inherent challenge of matching one code-pattern to many targets, as well as providing CI for all these different schedulers. I have come to the belief that ClusterManagers.jl should have ignored the question of requesting resources and instead should have focused on creating a Distributed.jl cluster within an allocation.
MPIClusterManagers.jl was spun out of MPI.jl and mostly ignores the question of requesting resources and using MPI within an allocation to create a Distributed.jl cluster.
Additional maintainers or competing packages to ClusterManagers would be welcome, and I particularly believe that Dagger.jl (cc: @jpsamaroo) needs support for cluster schedulers.
---
### Author:  [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)
#### Post date: [February 11, 2024, 8:14pm UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/16 "2024-02-11T20:14:40Z")
I’m not sure exactly what your requirements are, but perhaps this old post of mine might be helpful:
> [@Running Julia in a SLURM Cluster](https://discourse.julialang.org/t/running-julia-in-a-slurm-cluster/67614/2):
>
> I recently set up some scripts for running Julia jobs on a Slurm cluster. All of my jobs just use a single node with multiple CPUs. I’ll describe my approach, but I’m not an expert in HPC, so I’m not sure if everything that I’m doing is 100% correct. My approach is to write two scripts: a Slurm script and a Julia script. I currently am not using ClusterManagers. My mental model is that if I request one node with multiple CPUs, Slurm provisions a virtual machine with multiple cores, and Julia wi…
I no longer work with SLURM, so I can’t really answer any questions beyond that. 🙂
---
### Author:  [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)
#### Post date: [February 11, 2024, 9:19pm UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/17 "2024-02-11T21:19:32Z")
Thanks everyone. ~~(Sorry to nitpick here but it is perhaps a bit misrepresentative it is called “Distributed” despite only _itself_ handling multiprocessing on a single node — not what I would call distributed computing… but just a minor point and I digress)~~ Edit: I am completely wrong, see @aplavin’s helpful correction below!
@CameronBieganek I might not have been clear but I have been using ClusterManagers and Distributed quite extensively for maybe 3 years now. They are the current distributed computing backend for PySR and SymbolicRegression.jl. I have in the past been able to get decent communication going over ~4 nodes x 128 cores working with ClusterManagers.jl + Distributed.jl (your example has `--nodes=1` which is why you can get away with only Distributed.jl — it doesn’t need to interact with slurm).
It’s just they’ve always felt a bit hacky and unreliable, with users of my libraries complaining they couldn’t get it working on PBS clusters (turns out ClusterManagers has been broken for two years there, with no fixes on the horizon).
And now, Slurm clusters have broken too! Of course there’s always the option to fix everything myself but not exactly what I signed up for as a user 😪
---
### Author:  [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)
#### Post date: [February 11, 2024, 10:14pm UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/18 "2024-02-11T22:14:43Z")
At the very least I think the README could be more honest about the fact that most of the cluster managers it claims to support don’t actually work…
With its current state you get people like me spending weeks and weeks implementing support for it – thinking I will be able support all of these types of clusters – only to find out it’s actually unmaintained and broken. The README should just be honest about this, and point people to MPI.jl instead.
---
### Author:  [@mkitti](https://discourse.julialang.org/u/mkitti)
#### Post date: [February 11, 2024, 10:43pm UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/19 "2024-02-11T22:43:29Z")
My experience with clusters and ClusterManagers.jl is that it vastly depends on who is submitting and maintaining the recipes. Clusters vary by quite a bit. Perhaps the most consistency I have seen is within the DOE supercomputers, but even then there are variations between the clusters.
This mostly works for me, but that’s because @bjarthur put together the LSF support. I can physically locate him with a short walk or a Zoom call.
My best advice for this is to get your cluster admins involved. I have to do the same thing for Dask, Spark, and Ray deployments. For the most part cluster admins seem to appreciation being asked rather than users trying to improvise a solution. The main time when I’ve seen this become a headache is when cluster administration has been outsourced to a third party.
---
### Author:  [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)
#### Post date: [February 11, 2024, 11:13pm UTC](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050/20 "2024-02-11T23:13:49Z")
Update – made a PR here: [Highlight lack of ongoing support on README by MilesCranmer · Pull Request #197 · JuliaParallel/ClusterManagers.jl · GitHub](https://github.com/JuliaParallel/ClusterManagers.jl/pull/197)
* * *
Thanks for the tip @mkitti, I can try. I should mention that most of my concern is about downstream users rather than myself. I can always hack together a working solution for my own cluster, but I am more interested in keeping my downstream users happy and able to use their clusters for SymbolicRegression.jl searches. For every barrier in the way (such as needing to email their sysadmin, etc) I will end up losing a significant fraction of potential users
[Next page](https://discourse.julialang.org/t/is-clustermanagers-jl-maintained-or-how-to-do-multi-node-calculations-in-julia/110050.md?page=2)