# Attaching workers to cores

**URL:** <https://discourse.julialang.org/t/attaching-workers-to-cores/43499>\
**Category:** General Usage\
**Tags:** distributed\
**Created:** [July 22, 2020, 3:02pm UTC](https://discourse.julialang.org/t/attaching-workers-to-cores/43499 "2020-07-22T15:02:40Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![nishaChandramoorthy](https://avatars.discourse-cdn.com/v4/letter/n/7993a0/32.png) [@nishaChandramoorthy](https://discourse.julialang.org/u/nishaChandramoorthy)\
**Post date:** [July 22, 2020, 3:02pm UTC](https://discourse.julialang.org/t/attaching-workers-to-cores/43499/1 "2020-07-22T15:02:40Z")

</div>

Hi,

I am trying to run a program using `Distributed`. The workflow is to have  
each remote worker execute the same function (with different arguments)  
that is called using `remotecall`.  
The problem is that initially all workers seem to be running on different cores  
but suddenly, all the execution switches to using a single core. Could you please  
help me keep the different workers on different logical cores?

Here is what I am doing:

```julia
using Distributed
addprocs(15)
@everywhere include("rijke_tangent_state_estimation.jl")
f = Array{Future,1}(undef, nworkers())
for i = 1:nworkers()
        filename = string("../data/rijke_exp3_", string(i), ".jld") 
        put!(RemoteChannel(i), filename) 
        @async f[i] = remotecall(assimilate_parameter_and_trajectory, i, filename)
        println("sent remote call to ", i)
end
for i = 1:nworkers()
        println(fetch(f[i]))
        println(myid(), "is done..")
end

```

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [July 22, 2020, 4:56pm UTC](https://discourse.julialang.org/t/attaching-workers-to-cores/43499/2 "2020-07-22T16:56:46Z")

</div>

How quick does your function run? Does using `pmap` instead make more sense?

---

<div class="post-metadata">

**Author:** ![nishaChandramoorthy](https://avatars.discourse-cdn.com/v4/letter/n/7993a0/32.png) [@nishaChandramoorthy](https://discourse.julialang.org/u/nishaChandramoorthy)\
**Post date:** [July 22, 2020, 6:13pm UTC](https://discourse.julialang.org/t/attaching-workers-to-cores/43499/3 "2020-07-22T18:13:00Z")

</div>

It’s an expensive function, each evaluation takes about 3 hours on a single core i7 1.8GHz CPU.  
The real pressing question is why does the above code switch to a single core evaluation, given that, when it started, it was using up as many cores as workers?  
Thanks for letting me know about pmap! I can try it and let you know.  
Does `pmap` have a different behavior for parallelism?

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [July 22, 2020, 6:24pm UTC](https://discourse.julialang.org/t/attaching-workers-to-cores/43499/4 "2020-07-22T18:24:49Z")

</div>

I don’t know what the internals are like for `pmap`, but the idea is that it automatically takes care of the scheduling for the long running functions.

---

<div class="post-metadata">

**Author:** ![nishaChandramoorthy](https://avatars.discourse-cdn.com/v4/letter/n/7993a0/32.png) [@nishaChandramoorthy](https://discourse.julialang.org/u/nishaChandramoorthy)\
**Post date:** [July 22, 2020, 6:27pm UTC](https://discourse.julialang.org/t/attaching-workers-to-cores/43499/5 "2020-07-22T18:27:19Z")

</div>

Thanks, I just tried `pmap` and it ends up doing the same thing.  
For the first few seconds, 16 cores are being used, and afterward, only one is being used.

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [July 22, 2020, 6:29pm UTC](https://discourse.julialang.org/t/attaching-workers-to-cores/43499/6 "2020-07-22T18:29:28Z")

</div>

It could be that your functions are returning quicker than you thought. Are you sure the function takes about 3 hours? Can you post more code and also how you called `pmap`? Thanks

---

<div class="post-metadata">

**Author:** ![nishaChandramoorthy](https://avatars.discourse-cdn.com/v4/letter/n/7993a0/32.png) [@nishaChandramoorthy](https://discourse.julialang.org/u/nishaChandramoorthy)\
**Post date:** [July 22, 2020, 6:56pm UTC](https://discourse.julialang.org/t/attaching-workers-to-cores/43499/7 "2020-07-22T18:56:44Z")

</div>

Thanks for responding!

I am sure because I tried the function serially and it works perfectly fine.  
It takes as long as I mentioned.  
The problem is definitely in how I am parallelizing. Here is how I use `pmap`:

```julia
using Distributed
addprocs(15)
@everywhere include("rijke_tangent_state_estimation.jl")
filenames = Array{String,1}(undef, nworkers())
for i = 1:nworkers()
        filenames[i] = string("../data/rijke_exp3_", string(i), ".jld")
end
pmap(assimilate_parameter_and_trajectory, filenames)

```

---

<div class="post-metadata">

**Author:** ![nishaChandramoorthy](https://avatars.discourse-cdn.com/v4/letter/n/7993a0/32.png) [@nishaChandramoorthy](https://discourse.julialang.org/u/nishaChandramoorthy)\
**Post date:** [July 22, 2020, 7:03pm UTC](https://discourse.julialang.org/t/attaching-workers-to-cores/43499/8 "2020-07-22T19:03:19Z")

</div>

OK I noticed that with `pmap` or using `remotecall` as in my original post, the code stops with a segmentation fault after a while. This is due to an out-of-memory exception which should not occur because the function does allocate memory less than the memory per core of my cluster node.  
However, if all 16 workers run on the same core, which is what was happening, I would not be surprised if there is a seg fault due to an out of memory error.

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [July 22, 2020, 7:06pm UTC](https://discourse.julialang.org/t/attaching-workers-to-cores/43499/9 "2020-07-22T19:06:57Z")

</div>

Yes, that was my next question. Make sure there is enough memory for each julia worker process running the code. Furthermore, all the results from `pmap` are returned back to the head node (or the main core it was run on) so you must make sure there is available memory there as well to collect the results.

For example, my head node where I launch Julia and submit my `pmap` code only has 32 gb of memory. My compute nodes have 128gb of memory. So I have to make sure that when my workers still the objects back to `pmap`, it’s less than 32 gb in memory.

---

<div class="post-metadata">

**Author:** ![nishaChandramoorthy](https://avatars.discourse-cdn.com/v4/letter/n/7993a0/32.png) [@nishaChandramoorthy](https://discourse.julialang.org/u/nishaChandramoorthy)\
**Post date:** [July 22, 2020, 7:27pm UTC](https://discourse.julialang.org/t/attaching-workers-to-cores/43499/10 "2020-07-22T19:27:22Z")

</div>

Thank you so much @affans!  
This was super useful:

> [@affans](#):
>
> Furthermore, all the results from `pmap` are returned back to the head node (or the main core it was run on) so you must make sure there is available memory there as well to collect the results.

I removed the return values of the function (just wrote the arrays to disk)  
and this seemed to fix everything. I think returning all the arrays at once made the main core go out of RAM. Thanks for your help! 🙂

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [July 22, 2020, 8:09pm UTC](https://discourse.julialang.org/t/attaching-workers-to-cores/43499/11 "2020-07-22T20:09:29Z")

</div>

In HPC the normal method to pin processes is to use the numactl tools (linux specific).  
The slurm scheduler can do this.  
This actually leads me to ask how is process pinning accomplished in Julia? You do ask a good question!

As you say it is important. Also if processes switch between cores it can cause cache invalidation.