Memory trouble with @distributed

New to Julia
2

This seems to be feature of SharedArrays, see this more minimal example:

using SharedArrays

function consumes_mem()
    shared_var = SharedArray{Float64,2}(1000,100000,init=0.0)
    for i=1:1000 # Memory use seems to grow every time this loop is called
        random_num = randn(100000) 
        @. shared_var[i,:] = random_num
    end
    result = sum(shared_var)
    return result
end 

for i=1:100
	println(string(consumes_mem()))
end

which reproduces the memory consumption. (by the way, your code has an error, it’s: @. shared_var[i,:] = random_num see the row/col indices)

Why I say feature? Because:

help?> SharedArray
...
The shared array is valid as long as a reference to the SharedArray object exists on the node which created the mapping.
...

Of course, in above case it shouldn’t outlive the function call.
There is an existing issue: Memory Usage and SharedArrays · Issue #33990 · JuliaLang/julia · GitHub Wrong issue, but there Issues · JuliaLang/julia · GitHub are several related issues. Perhaps another specific one wouldn’t harm?

But, for a work around, if it is somehow a real problem to solve, our original code should better look like:

using Distributed
using SharedArrays
if nprocs() == 1
    addprocs(4)
end

function main()
    shared_var = SharedArray{Float64,2}(1000,100000,init=0.0); 
    for i=1:100
        r=consumes_mem(shared_var)
		println(string(i)*": "*string(r))
    end
end

@everywhere function consumes_mem(shared_var)
    @sync @distributed for i=1:1000 # Memory use seems to grow every time this loop is called
        random_num = randn(100000) 
        @. shared_var[i,:] = random_num
    end
    result = sum(shared_var)
    return result
end 

main()