# Dagger not fully utilizing CPU cores

**URL:** <https://discourse.julialang.org/t/dagger-not-fully-utilizing-cpu-cores/131733>\
**Category:** Performance\
**Created:** [August 20, 2025, 7:13am UTC](https://discourse.julialang.org/t/dagger-not-fully-utilizing-cpu-cores/131733 "2025-08-20T07:13:27Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Miroboru](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/miroboru/32/47659_2.png) [@Miroboru](https://discourse.julialang.org/u/Miroboru)\
**Post date:** [August 20, 2025, 7:13am UTC](https://discourse.julialang.org/t/dagger-not-fully-utilizing-cpu-cores/131733/1 "2025-08-20T07:13:27Z")

</div>

I have a time consuming simulation that is run over a set of signal to noise ratios, and thus is trivial to parallelize. I am currently using Dagger.jl to start 24 processes (I am on a 24 core threadripper system with 256 GB of RAM), and is running the simulations like below:

```julia-auto
simnum = 1000
numsnrs = 10
snrs = LinRange(0.0, 40.0, numsnrs)
dtasks = map(enumerate(snrs)) do (n,snr)
    newsp = @set simpar.SNR=snr
    Dagger.@spawn sim_fun(sim, newsp, simnum)
end
errs = fetch.(dtasks)

```

What happens is the following - if I set numsnrs to 24, it will only use eg. 16 cores, then six and then two. If I set numsnrs to 12, it will first run on ten cores and then two. For numsnrs=10 I see it run on eight, then two cores.

All in all there are plenty of resources on the system. RAM usage is below 20%, and each process that is actually running is at 99-100% and no more, meaning there is no underlying parallelization in eg. FFTs or other low level computations.

Anyone have any ideas on what may be going on here? My simulations are taking at least twice as long as they should, which is not ideal to say the least.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 20, 2025, 1:46pm UTC](https://discourse.julialang.org/t/dagger-not-fully-utilizing-cpu-cores/131733/2 "2025-08-20T13:46:59Z")

</div>

have you tried [GitHub - JuliaFolds2/OhMyThreads.jl: Simple multithreading in julia](https://github.com/JuliaFolds2/OhMyThreads.jl/)

---

<div class="post-metadata">

**Author:** ![Miroboru](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/miroboru/32/47659_2.png) [@Miroboru](https://discourse.julialang.org/u/Miroboru)\
**Post date:** [August 20, 2025, 2:14pm UTC](https://discourse.julialang.org/t/dagger-not-fully-utilizing-cpu-cores/131733/3 "2025-08-20T14:14:48Z")

</div>

Thanks for the suggestion, but from what I can see it is threads only, and I need to use processes to avoid thrashing the garbage collector (I could probably rewrite parts of my simulation to use preallocated arrays, but for now I just use processes and avoid the worst of the problem)

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 20, 2025, 2:33pm UTC](https://discourse.julialang.org/t/dagger-not-fully-utilizing-cpu-cores/131733/4 "2025-08-20T14:33:41Z")

</div>

> [@Miroboru](#):
>
> avoid thrashing the garbage collector

we have a concurrent marking and parallel sweeping GC now: [Multi-Threading · The Julia Language](https://docs.julialang.org/en/v1/manual/multi-threading/#Multiple-GC-Threads) set this to something like `4,1`

give it a try

---

<div class="post-metadata">

**Author:** ![Miroboru](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/miroboru/32/47659_2.png) [@Miroboru](https://discourse.julialang.org/u/Miroboru)\
**Post date:** [August 21, 2025, 10:48am UTC](https://discourse.julialang.org/t/dagger-not-fully-utilizing-cpu-cores/131733/5 "2025-08-21T10:48:52Z")

</div>

I tried using OhMyThreads, and while it is great and something I will keep at hand for later use, the result was like expected, with 30-40% time spent in garbage collection, versus \<10% for a single thread simulation.

Unfortunately there is not much I can do about the memory usage - most of it are return arrays from DSPs xcorr and resample, both of which do not do in-place calculations.

I will dig a little deeper into what Dagger is doing for now, as multiprocessing seems to be the way to go for this particular problem.

---

<div class="post-metadata">

**Author:** ![Miroboru](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/miroboru/32/47659_2.png) [@Miroboru](https://discourse.julialang.org/u/Miroboru)\
**Post date:** [August 21, 2025, 1:05pm UTC](https://discourse.julialang.org/t/dagger-not-fully-utilizing-cpu-cores/131733/6 "2025-08-21T13:05:43Z")

</div>

So I found a solution that works in my case at least. One can force the task to be executed on a specific worker by using scopes. At least when the number of jobs is smaller than the number of processes, this hack works:

```julia-auto
snrs = LinRange(0.0, 30.0, 12)
simnum = 1000
dtasks = map(enumerate(snrs)) do (n,snr)
    newsp = @set simpar.SNR=snr
    scp = Dagger.scope(worker=n+1)
    Dagger.@spawn scope=scp sim_doa(newsp, simnum)
end
@time errs = fetch.(dtasks)
;

```

I guess an even better approach would be to increase the granularity of the simulation, eg. by splitting each SNR point into multiple smaller samples and then combining them.