Initializing local variables on workers without using excessive memory

Specific Domains Julia at Scale
, ,
1

I’m using Julia and JuMP to solve a massive optimization problem with decomposition on a HPC. For this purpose, I want to initialize an independent optimization problem on each worker and then solve it repeatedly. So far, my setup to initialize the local problem on each worker corresponds to the minimum example below. In the actual problem, b is large and requires substantial resources to create, so it’s a step that needs to be done locally.

using Distributed
addprocs(2)

# initialize function on worker
@everywhere begin 
    testFunc(b) = b * 2
end

# initialize local variable on each worker
w = []
for x in workers()
    ini = @async @everywhere x begin
        b = myid()
    end
    push!(w, ini)
end
wait.(w)

# call function on worker
fetch(Distributed.@spawnat 3 testFunc(b))

This setup works on small-scale examples, but the initialization step requires excessive memory. When I benchmarked the initialization step for a large-scale example separately without distributed computing, it never required more than 22 GB. However, in the distributed case, my workers always run out of memory and get killed, although they have 32 GB available, and I use the heap-size-hint flag with a value of 28 GB.

I noticed that the initialization does not run out of memory if I replace @async @everywhere in the example above with @spawnat. However, the variable seems to be initialized in the wrong scope then, and the subsequent function call fails. Yet, it shows that the problem is not with the memory allocation for the workers.

2

I figured, I can use @spawnat but have to make b a global variable, so it is accessible outside of the function scope. I’m still curios why @everywhere combined with a specific worker allocation uses so much memory.

using Distributed
addprocs(2)

# initialize function on worker
@everywhere begin 
    testFunc(b) = b * 2
end

# initialize local variable on each worker
w = []
for x in workers()
    ini = @spawnat x begin
        global b = myid()
    end
    push!(w, ini)
end
wait.(w)

# call function on worker
fetch(Distributed.@spawnat 3 testFunc(b))