Why does julia use thousands of cpu-hours to compute 1+2?

General Usage
,
1

I’m fairly new to julia, and not overly experienced with scaling to huge numbers of processes, so forgive me if I’m missing something obvious.

Alright, to be honest, I actually used a couple thousand cpu-hours to run using DoubleFloats, but it looks like 1+2 would have done the same. From my testing, the time taken to call

@everywhere <basically any fast piece of code>

scales linearly with the number of processes (at least the first time it’s called), and the constant in front of that scaling is ~1 second, even for something as simple as

@everywhere 1+2

But that linear wallclock scaling means that the cpu-time spent on this call scales quadratically. Now, that ~1 second may not sound like much, but it can easily get to ~1 hour (wallclock) with a few thousand processes — which is thousands of cpu-hours.

Obviously I run into this mostly with @everywhere using WhateverPackage, where the package can be something as simple as Base. But it doesn’t matter whether DEPOT_PATH points to a local SSD for each worker or some slow home directory.

I didn’t realize this was going to happen, saw that my code worked nicely on a few dozen processes, and threw it onto a few thousand processes (several dozen nodes with 56 cores each, and one process per core), which took about an hour to get past the first @everywhere using DoubleFloats, thereby wasting a couple thousand cpu-hours on the first occurrence of @everywhere!!! I can’t really afford to waste this much compute time for each run, so this makes julia unusable for what I need here. (Making up the difference with threading isn’t really in the cards.)

Am I missing something? Can I prime my workers to know how to use @everywhere or distribute this preparation so that it doesn’t go quadratically? Can I somehow use pmap and friends without @everywhere using SomePackage? Or is this just somewhere julia won’t be able to reach?

4 Likes
2

See also here and here.

3

I’ve hacked together a precompilation script here, which produces a sysimage with all the necessary Distributed code precompiled. This drastically reduces the time taken to call @everywhere 1+2. I’ve tested it on 100 processes, and the time goes down from ~70 seconds to ~2 seconds. If this scales up to thousands of processes, it’s definitely something I can handle.

8 Likes
4

Can you try reverting https://github.com/JuliaLang/julia/pull/37816 and see how it changes things?

1 Like
5

Hm. Weird results. It is true that the parent (2f6508d) of that PR’s merge commit is significantly faster to call @everywhere 1+2 on 100 processes: 13.7s, as opposed to 52.7s after the merge (a4bfb9ce14).

It’s interesting that the slower time from that old code is still noticeably faster than v1.6.2 without any extra precompilation. But also, if I modify the parent commit to also precompile @everywhere 1+2, it doesn’t improve the time significantly; it’s still nowhere near the ~2 seconds I get from precompiling that on v1.6.2.

However, if I check out current master and try to reverse that patch manually, it doesn’t make any difference at all. I notice that toward the end of the build, it reports

Executing precompile statements... 1209/1242

whether it’s fresh master or I remove those comments you put in with that PR. I guess Distributed === nothing at that point in a fresh build nowadays?

6

It shouldn’t be, did you check it? You can just print it out.

7

Oh you’re right. It’s definitely something at that point, and precompile_script is definitely getting that new bit of code. Still, it doesn’t change the numbers being reported at the end of the build process.

8

Thanks for noticing this. Try with ensure that the non-REPL precompile code actually get precompiled by KristofferC · Pull Request #41898 · JuliaLang/julia · GitHub.

1 Like
10

That does seem to do something. With your PR, I see

Executing precompile statements... 1411/1447

which is up from 1368/1404 on the parent commit.

However, if I now re-enable that bit of code you disabled in #37816, it gives me this (complete with ugly formatting):

Generating REPL precompile statements... 31/31
Executing precompile statements... 176/1602┌ Warning: Inactive precompile statement
│   form = Tuple{Distributed.var"#@distributed", LineNumberNode, Module, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any, Any}
└ @ nothing nothing:0
Executing precompile statements... 1564/1602

(If I also add @everywhere 1+2, I get just that same warning, as well as 1588/1626.) I guess that increase means that a lot of what @distributed needs is getting compiled despite the warning. :man_shrugging:

11

Yeah, warnings like that are not unexpected.

1 Like