# Parallelize (pre) compilation?

**URL:** <https://discourse.julialang.org/t/parallelize-pre-compilation/46087>\
**Category:** Performance\
**Created:** [September 5, 2020, 3:15am UTC](https://discourse.julialang.org/t/parallelize-pre-compilation/46087 "2020-09-05T03:15:54Z")\
**Posts on this page:** 1\
**Showing post:** 9

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [September 6, 2020, 2:29pm UTC](https://discourse.julialang.org/t/parallelize-pre-compilation/46087/9 "2020-09-06T14:29:55Z")

</div>

> [@Samuel\_Ainsworth](#):
>
> AFAICT this process is completely single-threaded. This seems like a bit of a shame.

Yes it is, the precompilation is serial (while there’s a loophole to allow precompiling in parallel, see below), and I’ve been looking into the loading part (even with precompilation done) it is also serial, meaning single-threaded.

> [@Parallel loading of packages, to help e.g. Queryverse (VegaLite), time-to-first plot](https://discourse.julialang.org/t/parallel-loading-of-packages-to-help-e-g-queryverse-vegalite-time-to-first-plot/45700):
>
> First I want to say, time-to-first-plot isn’t really an issue (I consider it a solved problem if you know what you’re doing, using right packages/options), e.g. in about a sec. on Julia 1.6 (you can excpect this speed soon, on default options in 1.6): $ julia -O1 --compile=min -q julia\> @time using VegaLite 0.902355 seconds (1.20 M allocations: 83.260 MiB) julia\> @time @vlplot( data={ values=[ {a="A",b=28},{a="B",b=55},{a="C",b=43}, …

I’m personally less worried about the precompilation part, as it only happens once, I think more about the general loading phase, and note, it also involves some compilation (precompilation is only partial).

If you read my that other thread of mine to the end, no, I’m not “underestimating” the Julia developers, this is just a difficult problem, and other stuff may have had priority. I want to say @kristoffer.carlsson (and the other Julia developers) are doing a great job, also with answering.

The problem as I see it, is, in Julia:

```julia
using A
using B

```

is not the same (in general, could have side-effects, and other issues I’m not getting into here) as even (let alone, doing both at the same time, Julia’s semantics disallow that, currently):

```julia
using B
using A

```

[I’ve shown in another thread, that the time for doing the above can differ, so it’s hard to know the best order of `using packages, with n! possible orderings.]

I did a test, to make sure precompilation CAN happen in parallel, as I thought (given separate Julia processes/terminals). This might at least be helpful to know, if you anticipate compiling packages, like these package below that are big.

You can do the downloading of separate packages, in parallel, obviously, in separate terminals, and also then the precompiling:

```julia
julia> @time using Plots
[Info: Precompiling Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]
134.201107 seconds (6.86 M allocations: 478.701 MiB, 0.29% gc time)

```

in the other terminal:

```julia
julia> @time using Gtk
[Info: Precompiling Gtk [4c0ca9eb-093a-5379-98c5-f87ac0bbbf44]
Gtk-Message: 14:08:49.532: Failed to load module "canberra-gtk-module"
Gtk-Message: 14:08:49.532: Failed to load module "canberra-gtk-module"
108.263449 seconds (3.77 M allocations: 245.317 MiB, 0.08% gc time)

```

and I confirmed doing both in the same terminal/Julia process takes the combined time, not only 134 sec.

So why is this? While Julia is great for parallel programming, by default, it has serial semantics, even for `using` (and it needs to preserve that illusion at least).

I believe `using A, B` has exactly the same semantics as `using A; using B`, i.e. also disallowing parallel semantics, while maybe that could be relaxed. You can also foresee the Julia runtime looking ahead, and seeing the `using B` in the latter case, and starting with it, what it could in theory parallelize, e.g. its compilation, if e.g. waiting for precompiling A.

---

_[View the full topic](https://discourse.julialang.org/t/parallelize-pre-compilation/46087)._