Hi all, Problem: There is a heavy_computation() which returns a large Array. The computation of the heavy_computation() is parallelized with the help of Dagger.jl. The DTasks are spawned and later fetched. Running this code results in an ever growing memory usage (3GB-4GB). It seems to me that th…

Dagger’s usage of remote references (in the form of MemPool.DRef, which all tasks generate) causes the GC to be a bit slower to free memory than it normally would be. You can ask Dagger to more aggressively free memory by doing Dagger.MemPool.MEM_RESERVED[] = 1024^3 to reserve 1GB of memory (amount …

Thanks for your reply! I think I nailed it down to two problems, let me share some data and code to elaborate on them. Problem 1: Garbage collection, if activated does not properly clean the memory! As suggested, I added Dagger.MemPool.MEM_RESERVED[] = 10^6 * 1000 # 1000 MB Dagger.MemPool.MEM_…

[image] Rutger0000: The way of determining free_memory() can have unpredictable behavior on a SLURM node. If you think that is an issue with Dagger.jl, can you create an issue at GitHub · Where software is built ?

Thanks for the excellent analysis! As you’ve found, the process of detecting remaining memory is OS-specific, because the built-in method for detecting remaining memory (Sys.maxrss()) has not, in previous testing, reported a value appropriate for this purpose (I don’t recall specifics, but I wouldn…

Thanks for the reply and I am happy to investigate further. From what I can tell so far, the two problems likely require different solutions. Problem 1: The garbage collector does not effectively clean up the objects. What would you suggest to further investigate this? Monitor Refcounts in MemPo…

DArray: Memory cannot GC when the variables are reassigned, causing `Out of Memory` error or halting problem opened 03:35PM - 10 Aug 25 UTC (UTC) [image] islent …

Memory usage & garbage collection of Dagger

General Usage

ufechner7 October 22, 2025, 3:58pm 4

If you think that is an issue with Dagger.jl, can you create an issue at GitHub · Where software is built ?

Topic		Replies	Views
Garbage collection not aggressive enough on Slurm Cluster Julia at Scale multithreading , garbage-collection	17	3040	August 4, 2021
Multithreaded memory usage General Usage	19	1951	July 14, 2023
Poor performance of garbage collection in multi-threaded application Julia at Scale garbage-collection	22	5678	February 3, 2022
Dagger not fully utilizing CPU cores Performance	5	247	August 21, 2025
Some Dagger.jl beginner questions Julia at Scale dagger	6	805	August 23, 2023

Memory usage & garbage collection of Dagger

Related topics