Is there anything in the Julia ecosystem similar to GitHub - Libensemble/libensemble: A Python toolkit for coordinating asynchronous and dynamic ensembles of calculations. ?
I’m not familiar with libensemble, but glancing through the README, the closest thing I’m aware of might be some combination of Dagger.jl and the stdlib Distributed.jl might get you some of the way there.
Yeah, this looks like a more cumbersome alternative to Dagger.jl. Dagger doesn’t make you work as hard to spawn and manage a large number of tasks, and you can run it under MPI for lots of parallelism (helpers for this coming soon).
2 Likes
Thank you for the answers. Does Dagger.jl integrate nicely with the SLURM job scheduler? That is a must for any HPC application.
For what is worth, not all HPC systems use SLURM.
Dagger.jl composes with packages like ClusterManagers.jl or (I assume) SlurmClusterManager.jl.
A typical pattern in Julia is that packages tend to be small and focused, and work together to make bigger things possible, rather than having lots of functionality wrapped up in a single package.