Heap-size-hint in MPIClusterManager.jl for out of memory problems

General Usage
, ,
1

Hi, for the MPI transport option, where only MPI can be used for communication on the cluster, I’ve gotten the example from MPIClusterManagers.jl working well for my use case. However, I would like to add a heap size hint, because for very long runs, I run into out of memory problems. For addprocs(), this can be done with something like

addprocs(960, exeflags=`--heap-size-hint=3G`) # 3GB per worker hint

but how in the MPI transport case, do I do this as there is no explicit addprocs()?

My setup is to call within qsub on PBS
mpirun -np 960 myprogram.jl

where myprogram.jl is of the form

using MPIClusterManagers, Distributed
import MPI
MPI.Init()
rank = MPI.Comm_rank(MPI.COMM_WORLD)
sz = MPI.Comm_size(MPI.COMM_WORLD)
if rank == 0
    @info "size is $sz"
end
manager = MPIClusterManagers.start_main_loop(MPI_TRANSPORT_ALL)
@info "there are $(nworkers()) workers"
include("01_read_data.jl")
include("02_set_options.jl")
## Do stuff as normal with Distributed
@everywhere using Distributed
@everywhere using MyModule
# @sync for w in workers()
#   @async remotecall_fetch(myfunc, w, myopts...) # returns nothing on completion
# end
## exit gracefully
MPIClusterManagers.stop_main_loop(manager)
rmprocs(workers())
exit()

Many thanks for any hints or experience with this situation.

2

This won’t work. You forgot the julia executable, to which you can then also pass the command line option.

3

Apologies, you are right, I meant, this:

mpirun -np 960 julia --heap-size-hint=3GB myprogram.jl

This error notwithstanding, I’m looking at https://github.com/JuliaLang/julia/issues/50673, about the interaction between addprocs() and the workers with respect to heap-size-hint. I don’t think the new aggressive gc on Julia 1.10.5 is enough for my long job, and the hint environment variable specified at the command line is valid only for the manager process.

How to pass this heap-size-hint environment variable to the workers when not using addprocs()?

4

There is no “manager process” when using MPI. All MPI processes are equal. (This is in contrast to Distributed.jl and addprocs.)

If using --heap-size-hint=3GB isn’t enough than you probably need to put fewer MPI ranks on a node.

UPDATE: I’m sorry, I didn’t quite read your post and didn’t notice that you want to combine MPI and Distributed. I don’t have much experience with this, I’m afraid.