I’m trying to parallelize a piece of code using Distributed, but it’s slowly eating up all my memory. Here’s an example: using Distributed using SharedArrays if nprocs() == 1 addprocs(4) end function main() for i=1:100 consumes_mem() end end function consumes_mem() shared_…

This seems to be feature of SharedArrays, see this more minimal example: using SharedArrays function consumes_mem() shared_var = SharedArray{Float64,2}(1000,100000,init=0.0) for i=1:1000 # Memory use seems to grow every time this loop is called random_num = randn(100000) @…

Memory trouble with @distributed

New to Julia

thomas.frederikse June 8, 2021, 5:13pm 5

Yes, agree! The need came from reading netCDF files in parallel, with each slice of the SharedArray coming from a different file. But this solved the issue.

Topic		Replies	Views
Memory Usage and SharedArrays Julia at Scale parallel , memory-allocation , distributed	8	1459	December 6, 2019
How to GC a SharedArray General Usage distributed , sharedarrays	6	1238	October 28, 2020
Mmap problem with SharedArrays Specific Domains first-steps , parallel , distributed	2	419	November 27, 2019
Using Distributed: computational efficiency Julia at Scale	5	971	June 24, 2019
Memory allocation and SharedArrays Julia at Scale question	3	757	August 7, 2019

Memory trouble with @distributed

Related topics