# Error when using "too many" workers

**URL:** <https://discourse.julialang.org/t/error-when-using-too-many-workers/10413>\
**Category:** Julia at Scale\
**Created:** [April 18, 2018, 2:20pm UTC](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413 "2018-04-18T14:20:06Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![bisraelsen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bisraelsen/32/3974_2.png) [@bisraelsen](https://discourse.julialang.org/u/bisraelsen)\
**Post date:** [April 18, 2018, 2:20pm UTC](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413/1 "2018-04-18T14:20:06Z")

</div>

Hi, I have been running code in parallel using `julia -p200` for a while. Now I am using a cloud server that has:

```bash
CPU(s) 96
Cores Per Socket 24
Sockets 2

```

By my calculations I should be able to run 96x24x2=4608 threads. However when trying to run something like `julia -p750` (I can run `julia -p500`) for example I get the following error:

```julia
brett_israelsen@instance-1:~/GitProjects/self_confidence/road_net$ julia -p4600
ERROR (unhandled task failure): pipe_link: too many open files (EMFILE)
Stacktrace:
 [1] setup_stdio(::Base.##374#375{Cmd}, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./process.jl:497
 [2] #spawn#373(::Nullable{Base.ProcessChain}, ::Function, ::Cmd, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./process.jl:511
 [3] (::Base.#kw##spawn)(::Array{Any,1}, ::Base.#spawn, ::Cmd, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./<missing>:0
 [4] #spawn#370(::Nullable{Base.ProcessChain}, ::Function, ::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./process.jl:392
 [5] spawn(::Base.CmdRedirect, ::Tuple{Base.DevNullStream,Pipe,Base.TTY}) at ./process.jl:392
 [6] (::Base.Distributed.##31#34{Base.Distributed.LocalManager,Dict{Any,Any},Array{WorkerConfig,1},Condition})() at ./event.jl:73
Master process (id 1) could not connect within 60.0 seconds.                                                    
exiting.
Worker 465 terminated.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
ERROR (unhandled task failure): Version read failed. Connection closed by peer.
Stacktrace:
 [1] (::Base.Distributed.##99#100{TCPSocket,TCPSocket,Bool})() at ./event.jl:73
Worker 464 terminated.
ERROR (unhandled task failure): Version read failed. Connection closed by peer.
Stacktrace:
 [1] (::Base.Distributed.##99#100{TCPSocket,TCPSocket,Bool})() at ./event.jl:73
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
Master process (id 1) could not connect within 60.0 seconds.
exiting.
.
.
.

```

I suppose that since I am fairly new to HPC, I may have unrealistic expectations, or could be doing something wrong. Hopefully someone can help me out, I would like to be able to use all of the available threads and am not sure how, or if I am already. Thanks!

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 18, 2018, 2:21pm UTC](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413/2 "2018-04-18T14:21:51Z")

</div>

> [@bisraelsen](#):
>
> By my calculations I should be able to run 96x24x2=4608 threads. However when trying to run something like julia -p750 (I can run julia -p500) for example I get the following error:

IIRC you need to use `addprocs` instead and set the topology to be something more sparse. @anon94023334 might know.

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [April 18, 2018, 2:31pm UTC](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413/3 "2018-04-18T14:31:56Z")

</div>

Judging from the error message you have a `ulimit` of 512 (check with `ulimit -a`). See [here](https://stackoverflow.com/questions/34588/how-do-i-change-the-number-of-open-files-limit-in-linux) for how to increase it (no clue if that works in your environment though).

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [April 18, 2018, 2:36pm UTC](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413/4 "2018-04-18T14:36:37Z")

</div>

Yup; I ran into this also. The default topology is “all-to-all”, which means, for N workers, you’re opening up N^2 files / sockets / whatever you’re using to communicate . If your workers don’t need to communicate with each other, consider using `:master_slave` instead (via `addprocs(N, topology=:master_slave)`).

IMO we should reconsider having `:all_to_all` as a default topology. I seem to recall a github issue on it.

---

<div class="post-metadata">

**Author:** ![bisraelsen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bisraelsen/32/3974_2.png) [@bisraelsen](https://discourse.julialang.org/u/bisraelsen)\
**Post date:** [April 18, 2018, 2:43pm UTC](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413/5 "2018-04-18T14:43:23Z")

</div>

For the sake of leaving a trail for future readers, this is the output of `ulimit -a`. I can’t really see the `512` you mentioned (unless is is the pipe size, which I doubt). However there is a file limit of `1024`, which might be related from the `too many open files` error.

```bash
brett_israelsen@instance-1:~/GitProjects/self_confidence/road_net$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 347860
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 347860
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

```

Looks like I can manually increase the limit for open files via `ulimit -n NEW_LIMIT`, but according to @anon94023334 using `:master_slave` might be the only practical solution based on the N^2 relationship.

---

<div class="post-metadata">

**Author:** ![bisraelsen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bisraelsen/32/3974_2.png) [@bisraelsen](https://discourse.julialang.org/u/bisraelsen)\
**Post date:** [April 18, 2018, 2:44pm UTC](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413/6 "2018-04-18T14:44:47Z")

</div>

So, does `addprocs(N, topology=:master_slave)` work in the same way as `julia -p N`? besides replacing the default `:all_to_all` mode?

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [April 18, 2018, 2:46pm UTC](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413/7 "2018-04-18T14:46:32Z")

</div>

Ah, ok. The 512 was just a guess, but I didn’t know about the N^2 relationship either. The relevant limit might be

```julia
pending signals (-i) 347860

```

then considering that 500^2=250000 \< 347860 \< 750^2 = 562500.

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [April 18, 2018, 2:48pm UTC](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413/8 "2018-04-18T14:48:42Z")

</div>

> [@bisraelsen](#):
>
> So, does `addprocs(N, topology=:master_slave)` work in the same way as `julia -p N`? besides replacing the default `:all_to_all` mode?

That’s my understanding.

---

<div class="post-metadata">

**Author:** ![markmont](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/markmont/32/3955_2.png) [@markmont](https://discourse.julialang.org/u/markmont)\
**Post date:** [April 18, 2018, 3:00pm UTC](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413/10 "2018-04-18T15:00:25Z")

</div>

Most cloud servers will have either two or four CPUs (chips). Each CPU chip resides in a single “socket” and has some number of cores on the chip. The total number of cores is therefore the number of CPUs times the number of cores per CPU. It sounds like you have two CPUs (sockets) with 24 cores per CPU, for a total of 48 cores; this would be a common, realistic, albeit somewhat high-end configuration for a cloud server.

For most types of CPUs (and, in particular, Intel Xeon CPUs) can support Hyperthreading. Hyperthreading allows each core to execute two threads (programs or different parts of the same program) at once, interleaved with each other. Hyperthreading usually speeds up performance, but there are some cases where it makes performance worse. On the system you describe, I suspect you have a total of 96 hypertheads on the 48 cores, and this is where the number “CPU(s) 96” comes from in your post above. Sometimes hyperthreads get referred to as “cores”, although this is not technically accurate; Amazon EC2 refers to hyperthreads as “vCPUs”.

So I suspect that what you really want to do is run “julia -p96”. Anything more may overload the cores with too many workers/threads, which will cause performance to suffer as each core attempts to switch between many tasks, which causes a large loss of efficiency due to the overhead of doing the switches.

One exception to what I say above is if you are using multiple clouds servers (nodes) together with MPI and a machinefile: in this case, you’ll have more cores available in total, but the different cloud servers will have to communicate with each other across the network, which adds some overhead. You will need to run your Julia code differently if this is the case; let us know if it is the case and if you’d like details.

Another exception to what I say above is if the cloud server you are using has one or more Intel Xeon Phi accelerator card in it (Knights Corner / Knight’s Landing). Having such a card would be very unusual, but can provide more cores on a single machine than would otherwise be possible.

I hope this helps.

---

<div class="post-metadata">

**Author:** ![bisraelsen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bisraelsen/32/3974_2.png) [@bisraelsen](https://discourse.julialang.org/u/bisraelsen)\
**Post date:** [April 18, 2018, 3:12pm UTC](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413/11 "2018-04-18T15:12:36Z")

</div>

> [@anon94023334](#):
>
> I seem to recall a github issue on it.

I believe this is the [issue](https://github.com/JuliaLang/julia/issues/21978) you are referring to.

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [April 18, 2018, 3:14pm UTC](https://discourse.julialang.org/t/error-when-using-too-many-workers/10413/12 "2018-04-18T15:14:59Z")

</div>

That’s the one, and [here’s some additional discussion.](https://github.com/JuliaLang/julia/pull/22814). My plea for backporting to 0.6 fell on deaf (or unsympathetic) ears.