# Heap-size-hint in MPIClusterManager.jl for out of memory problems

**URL:** <https://discourse.julialang.org/t/heap-size-hint-in-mpiclustermanager-jl-for-out-of-memory-problems/121050>\
**Category:** General Usage\
**Tags:** hpc, cluster, mpi\
**Created:** [October 8, 2024, 7:03am UTC](https://discourse.julialang.org/t/heap-size-hint-in-mpiclustermanager-jl-for-out-of-memory-problems/121050 "2024-10-08T07:03:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sparrowhawk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sparrowhawk/32/9148_2.png) [@sparrowhawk](https://discourse.julialang.org/u/sparrowhawk)\
**Post date:** [October 8, 2024, 7:03am UTC](https://discourse.julialang.org/t/heap-size-hint-in-mpiclustermanager-jl-for-out-of-memory-problems/121050/1 "2024-10-08T07:03:21Z")

</div>

Hi, for the MPI transport option, where only MPI can be used for communication on the cluster, I’ve gotten the [example](https://github.com/JuliaParallel/MPIClusterManagers.jl/blob/master/examples/cman-transport.jl) from MPIClusterManagers.jl working well for my use case. However, I would like to add a heap size hint, because for very long runs, I run into out of memory problems. For `addprocs()`, this can be done with something like

```julia
addprocs(960, exeflags=`--heap-size-hint=3G`) # 3GB per worker hint

```

but how in the MPI transport case, do I do this as there is no explicit `addprocs()`?

My setup is to call within `qsub` on PBS  
`mpirun -np 960 myprogram.jl`

where `myprogram.jl` is of the form

```julia
using MPIClusterManagers, Distributed
import MPI
MPI.Init()
rank = MPI.Comm_rank(MPI.COMM_WORLD)
sz = MPI.Comm_size(MPI.COMM_WORLD)
if rank == 0
    @info "size is $sz"
end
manager = MPIClusterManagers.start_main_loop(MPI_TRANSPORT_ALL)
@info "there are $(nworkers()) workers"
include("01_read_data.jl")
include("02_set_options.jl")
## Do stuff as normal with Distributed
@everywhere using Distributed
@everywhere using MyModule
# @sync for w in workers()
# @async remotecall_fetch(myfunc, w, myopts...) # returns nothing on completion
# end
## exit gracefully
MPIClusterManagers.stop_main_loop(manager)
rmprocs(workers())
exit()

```

Many thanks for any hints or experience with this situation.

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [October 8, 2024, 9:56am UTC](https://discourse.julialang.org/t/heap-size-hint-in-mpiclustermanager-jl-for-out-of-memory-problems/121050/2 "2024-10-08T09:56:14Z")

</div>

> [@sparrowhawk](#):
>
> My setup is to call within `qsub` on PBS  
> `mpirun -np 960 myprogram.jl`

This won’t work. You forgot the `julia` executable, to which you can then also pass the command line option.

---

<div class="post-metadata">

**Author:** ![sparrowhawk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sparrowhawk/32/9148_2.png) [@sparrowhawk](https://discourse.julialang.org/u/sparrowhawk)\
**Post date:** [October 8, 2024, 10:41am UTC](https://discourse.julialang.org/t/heap-size-hint-in-mpiclustermanager-jl-for-out-of-memory-problems/121050/3 "2024-10-08T10:41:09Z")

</div>

Apologies, you are right, I meant, this:

```julia
mpirun -np 960 julia --heap-size-hint=3GB myprogram.jl

```

This error notwithstanding, I’m looking at [https://github.com/JuliaLang/julia/issues/50673](https://github.com/JuliaLang/julia/issues/50673), about the interaction between `addprocs()` and the workers with respect to `heap-size-hint`. I don’t think the new aggressive gc on Julia 1.10.5 is enough for my long job, and the hint environment variable specified at the command line is valid only for the manager process.

How to pass this heap-size-hint environment variable to the workers when not using `addprocs()`?

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [October 8, 2024, 2:23pm UTC](https://discourse.julialang.org/t/heap-size-hint-in-mpiclustermanager-jl-for-out-of-memory-problems/121050/4 "2024-10-08T14:23:15Z")

</div>

> [@sparrowhawk](#):
>
> I don’t think the new aggressive gc on Julia 1.10.5 is enough for my long job, and the hint environment variable specified at the command line is valid only for the manager process.

There is no “manager process” when using MPI. All MPI processes are equal. (This is in contrast to Distributed.jl and `addprocs`.)

If using `--heap-size-hint=3GB` isn’t enough than you probably need to put fewer MPI ranks on a node.

UPDATE: I’m sorry, I didn’t quite read your post and didn’t notice that you want to combine MPI and Distributed. I don’t have much experience with this, I’m afraid.