I’m fairly new to julia, and not overly experienced with scaling to huge numbers of processes, so forgive me if I’m missing something obvious. Alright, to be honest, I actually used a couple thousand cpu-hours to run using DoubleFloats, but it looks like 1+2 would have done the same. From my testi…

I’ve hacked together a precompilation script here , which produces a sysimage with all the necessary Distributed code precompiled. This drastically reduces the time taken to call @everywhere 1+2. I’ve tested it on 100 processes, and the time goes down from ~70 seconds to ~2 seconds. If this scales…

Can you try reverting https://github.com/JuliaLang/julia/pull/37816 and see how it changes things?

Hm. Weird results. It is true that the parent (2f6508d) of that PR’s merge commit is significantly faster to call @everywhere 1+2 on 100 processes: 13.7s, as opposed to 52.7s after the merge (a4bfb9ce14). It’s interesting that the slower time from that old code is still noticeably faster than v1.…

[image] moble: . I guess Distributed === nothing at that point in a fresh build nowadays? It shouldn’t be, did you check it? You can just print it out.

Oh you’re right. It’s definitely something at that point, and precompile_script is definitely getting that new bit of code. Still, it doesn’t change the numbers being reported at the end of the build process.

[image] moble: Still, it doesn’t change the numbers being reported at the end of the build process. Thanks for noticing this. Try with ensure that the non-REPL precompile code actually get precompiled by KristofferC · Pull Request #41898 · JuliaLang/julia · GitHub .

That does seem to do something. With your PR, I see Executing precompile statements... 1411/1447 which is up from 1368/1404 on the parent commit. However, if I now re-enable that bit of code you disabled in #37816, it gives me this (complete with ugly formatting): Generating REPL precompile sta…

Yeah, warnings like that are not unexpected.

Why does julia use thousands of cpu-hours to compute 1+2?

General Usage

moble August 8, 2021, 10:02pm 2

Topic		Replies	Views
@everywhere takes a very long time when using a cluster General Usage	16	3330	February 11, 2022
First call latency in distributed code Performance distributed , latency	42	1828	October 21, 2022
Baffling addprocs() with @everywhere Julia at Scale	17	3754	April 24, 2018
Several questions on distributed computing from a beginner General Usage question , parallel-computing	21	915	May 28, 2024
Questions about getting started with parallel computing Julia at Scale	18	6074	June 22, 2019

Why does julia use thousands of cpu-hours to compute 1+2?

Related topics