# Help reducing compilation and inference time

**URL:** <https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630>\
**Category:** Performance\
**Tags:** compilation, latency\
**Created:** [July 4, 2024, 2:32pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630 "2024-07-04T14:32:51Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![fatteneder](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fatteneder/32/33991_2.png) [@fatteneder](https://discourse.julialang.org/u/fatteneder)\
**Post date:** [July 5, 2024, 8:06pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/41 "2024-07-05T20:06:27Z")

</div>

The way you read this graph is that the width of a bar corresponds to the relative time taken in the call. The total width of the graph is the total time, and so wider bars correspond to more time taken in the call. The depth (or here height) of a stack of bars corresponds to the number of nested calls.

So I think the question here is what are the two ‘deserts’ to the left and to the right of the two peaks where seemingly nothing happens? It seems like \>80% of the time is spent there.

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 5, 2024, 8:14pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/42 "2024-07-05T20:14:17Z")

</div>

> what are the two small ‘deserts’ to the left and to the right of the two peaks where seemingly nothing happens?

Great question. The left-hand-side stack consists of the following calls:

```julia
./task.jl:974, poptask [inlined]
./task.jl:983, wait [inlined]
./condition.jl:130, #wait#621 [inlined]
./condition.jl:125, wait [inlined]
./lock.jl:229, lock [inlined]
./condition.jl:78, lock [inlined]
./threadingconstructs.jl:373, #137 [inlined]

```

While the right-hand-side has the following calls:

```julia
./task.jl:974, poptask [inlined]
./task.jl:983, wait [inlined]
./task.jl:672, task_done_hook [inlined]

```

My guess is that they are calls from the profiler? And maybe Julia is doing something under the hood that is not captured in this flame graph

---

<div class="post-metadata">

**Author:** ![fatteneder](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fatteneder/32/33991_2.png) [@fatteneder](https://discourse.julialang.org/u/fatteneder)\
**Post date:** [July 5, 2024, 8:22pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/43 "2024-07-05T20:22:35Z")

</div>

That’s interesting.

To clarify earlier statements.

> [@osimonn](#):
>
> I was able to generate a flame graph of the code (compilation + execution)

Do you mean by compilation here just the precompilation happening when you start a fresh REPL and type `using ...`, or does this measure compilation of all pkgs in the environment (presumably in an empty depot), like i.e. `using Pkg; Pkg.precompile()`?

Also: I think you did not yet say which Julia version you are using. So what’s the output of `versioninfo`?

> [@osimonn](#):
>
> `-O0` shaves down the compilation time significantly. It now takes ~83 seconds for the first run.

Can you also generate a flamegraph when running with the `-O0` option? Would be nice for a comparison.

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 5, 2024, 8:32pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/44 "2024-07-05T20:32:01Z")

</div>

> [@fatteneder](#):
>
> Also: I think you did not yet say which Julia version you are using. So what’s the output of `versioninfo`?

Completely forgot about that 🙂

```julia
julia> versioninfo()
Julia Version 1.9.0
Commit 8e630552924 (2023-05-07 11:25 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 8 × Intel(R) Xeon(R) Gold 6338N CPU @ 2.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, icelake-server)
  Threads: 8 on 8 virtual cores
Environment:
  JULIA_NUM_THREADS = auto

```

> [@fatteneder](#):
>
> Do you mean by compilation here just the precompilation happening when you start a fresh REPL and type `using ...`, or does this measure compilation of all pkgs in the environment (presumably in an empty depot), like i.e. `using Pkg; Pkg.precompile()`?

I do the following (roughly):

```julia
time julia --project -e "using OurPackage; main("some_input_file")"

```

> [@fatteneder](#):
>
> Can you also generate a flamegraph when running with the `-O0` option? Would be nice for a comparison.

Sure. It will take me some time so I will post it in another reply…

---

<div class="post-metadata">

**Author:** ![fatteneder](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fatteneder/32/33991_2.png) [@fatteneder](https://discourse.julialang.org/u/fatteneder)\
**Post date:** [July 5, 2024, 8:41pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/45 "2024-07-05T20:41:17Z")

</div>

Another guess: Did you monitor RAM usage during compilation? Could it be that RAM is saturated and compilation slows down due to excessive swapping?

---

<div class="post-metadata">

**Author:** ![Tortar](https://avatars.discourse-cdn.com/v4/letter/t/6bbea6/32.png) [@Tortar](https://discourse.julialang.org/u/Tortar)\
**Post date:** [July 5, 2024, 8:42pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/46 "2024-07-05T20:42:56Z")

</div>

One thing you should do is to update to 1.10, compilation time should be reduced a bit I think with that

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 5, 2024, 8:44pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/47 "2024-07-05T20:44:13Z")

</div>

Nope. I updated our code to 1.10 and the latency got worse so we had to revert back to 1.9

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 5, 2024, 8:46pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/48 "2024-07-05T20:46:35Z")

</div>

RAM usage is stable, around 5 GBs during compilation out of 32 GBs.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [July 5, 2024, 8:59pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/49 "2024-07-05T20:59:13Z")

</div>

You cannot even download anything to initialize `.julia/compiled`?

Without any caching available in your CI, the only solution is to minimize compilation. You can play with optimizations on a module level with the following macro.

```julia
help?> Base.Experimental.@optlevel
  Experimental.@optlevel n::Int

  Set the optimization level (equivalent to the -O command line argument)
  for code in the current module. Submodules inherit the setting of their
  parent module.

  Supported values are 0, 1, 2, and 3.

  The effective optimization level is the minimum of that specified on the
  command line and in per-module settings. If a --min-optlevel value is
  set on the command line, that is enforced as a lower bound.

```

The key here is to minimize the types and methods involved. Move type parameters to fields when possible. Dispatch using `if` statements instead of using multiple dispatch.

Check your dependencies. Use `@time_imports` to see what dependencies are taking a while to load.

Check for method invalidation and recompilation. Use a small number of concrete types rather than abstract types. Use `SnoopCompile[Core]` and `@snoopr` to see if you are forcing anything to _recompile_.

You may even want to use the `--trace-compile` command line option as a sanity check to see what is compiling.

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 5, 2024, 9:13pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/50 "2024-07-05T21:13:50Z")

</div>

> [@mkitti](#):
>
> You cannot even download anything to initialize `.julia/compiled`?

Historically, we did have some compiled functions in `.julia/compiled`. However, it was causing major issues to other teams so we deliberately remove the directory’s content.

> [@mkitti](#):
>
> You can play with optimizations on a module level with the following macro.

That’s a cool idea but I am afraid that this will increase the overall CI/CD jobs because the runtime post-compilation will increase.

> [@mkitti](#):
>
> The key here is to minimize the types and methods involved. Move type parameters to fields when possible. Dispatch using `if` statements instead of using multiple dispatch.
> 
> Check your dependencies. Use `@time_imports` to see what dependencies are taking a while to load.
> 
> Check for method invalidation and recompilation. Use a small number of concrete types rather than abstract types. Use `SnoopCompile[Core]` and `@snoopr` to see if you are forcing anything to _recompile_.
> 
> You may even want to use the `--trace-compile` command line option as a sanity check to see what is compiling.

I hate to break your bubble but I have tried **all** of these suggestions for the past year and nothing worked hence this post.

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 5, 2024, 9:27pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/51 "2024-07-05T21:27:08Z")

</div>

> [@fatteneder](#):
>
> Can you also generate a flamegraph when running with the `-O0` option? Would be nice for a comparison.

This is the flame graph running `julia` (I don’t know why the graph shifted from my previous post):

 ![perf](https://global.discourse-cdn.com/julialang/original/3X/a/e/ae6e067f7404a3343e02ec96654e07638cf3cc62.png)  
This is the flame graph running `julia -O0`:  
 ![perf_O0](https://global.discourse-cdn.com/julialang/original/3X/a/d/ade8c9da1194f2a151cf50650c13436212685202.png)

Again, for both graphs, the left cluster is purely Julia “compiler” calls whereas the right cluster is where our code or `main` gets called.

---

<div class="post-metadata">

**Author:** ![fatteneder](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fatteneder/32/33991_2.png) [@fatteneder](https://discourse.julialang.org/u/fatteneder)\
**Post date:** [July 5, 2024, 9:56pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/52 "2024-07-05T21:56:14Z")

</div>

I am trying to understand that empty space.

I might be on the wrong track here, IDK, but it seems to me that this might be a pkgimage issue.  
At least that would be a simple theory that explains most things.

Assume your code fails to generate (or load) pkgimages, and instead has to compile all your dependencies at every start.  
It would explain why nothing shows up in most of the graph, because precompilation happens in separately spawned processes, one for each pkg. It would also explain the task related calls in flamegraph (`wait, lock`), and possibly explain why the first graph shifted, because of the order in which they are compiled might slightly vary for pkgs that do not share dependencies. (An alternative explanation might be that the right empty space in the very first plot was pkg compilation for the code that displays the graph; IIRC ProfileView uses Gtk3.jl or so)

* * *

> [@osimonn](#):
>
> Historically, we did have some compiled functions in `.julia/compiled`. However, it was causing major issues to other teams so we deliberately remove the directory’s content.

Again to clarify, do you in any way tinker with `~/.julia/compiled`?

And what is on your `LOAD_PATH` and `DEPOT_PATH`?

Assuming its pkgimage related, you could start julia with `JULIA_DEBUG=loading julia -e ...` and then study the generated logs. If there are a lot of rejection messages, then that’s a smoking gun.

---

<div class="post-metadata">

**Author:** ![fatteneder](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fatteneder/32/33991_2.png) [@fatteneder](https://discourse.julialang.org/u/fatteneder)\
**Post date:** [July 5, 2024, 10:00pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/53 "2024-07-05T22:00:52Z")

</div>

Could you also try to install OmniPackage.jl (in a separate env) and report how long installation and `using OminPackage` took (roughly)?

That’s a pkg used to benchmark the performance of pkgimages.  
With such a test we could gauge whether there is an issue with the julia installation, system setup or if its an issue with your pkg.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [July 5, 2024, 10:02pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/54 "2024-07-05T22:02:15Z")

</div>

> [@osimonn](#):
>
> Historically, we did have some compiled functions in `.julia/compiled`. However, it was causing major issues to other teams so we deliberately remove the directory’s content.

If this is possible and we could resolve those issues, I think there is room to make progress. Could you elaborate?

> [@osimonn](#):
>
> That’s a cool idea but I am afraid that this will increase the overall CI/CD jobs because the runtime post-compilation will increase.

Not necessarily. You can avoid compiler passes on code that is either rarely used or that you know that is relatively simple. In other words, there are circumstances where less compilation could result in the same code generation.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [July 5, 2024, 10:49pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/55 "2024-07-05T22:49:29Z")

</div>

> [@fatteneder](#):
>
> I am trying to understand that empty space.

On earlier (and possibly also recent) versions of Julia, starting Julia with multiple threads and profiling will result in large empty spaces. IIRC the empty space is usually `[width of thread 1 profile] x [N - # of threads]`. Hard to say if that’s the culprit here, but an easy way to check would be to ensure `Threads.nthreads() == 1` before profiling.

Aside from checking the number of threads, another idea is to visualize the inference flamegraph using [Snooping on inference: @snoopi\_deep · SnoopCompile](https://timholy.github.io/SnoopCompile.jl/dev/snoopi_deep/#Viewing-the-results). This may help fill in more of that empty space because it captures part of what the compiler is doing.

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 6, 2024, 1:14am UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/56 "2024-07-06T01:14:09Z")

</div>

> [@fatteneder](#):
>
> Again to clarify, do you in any way tinker with `~/.julia/compiled`?

In the results I presented so far, no. Don’t worry about this detail. It is not relevant for this discussion.

> [@fatteneder](#):
>
> Assuming its pkgimage related, you could start julia with `JULIA_DEBUG=loading julia -e ...` and then study the generated logs. If there are a lot of rejection messages, then that’s a smoking gun.

I tried it. No out-of-the-ordinary debug messages. Mostly packages or cache loaded.

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 6, 2024, 1:23am UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/57 "2024-07-06T01:23:11Z")

</div>

> [@ToucheSir](#):
>
> On earlier (and possibly also recent) versions of Julia, starting Julia with multiple threads and profiling will result in large empty spaces. IIRC the empty space is usually `[width of thread 1 profile] x [N - # of threads]`. Hard to say if that’s the culprit here, but an easy way to check would be to ensure `Threads.nthreads() == 1` before profiling.

I will generate new flame graphs with `-t 1`

> [@ToucheSir](#):
>
> Aside from checking the number of threads, another idea is to visualize the inference flamegraph using [Snooping on inference: @snoopi\_deep · SnoopCompile](https://timholy.github.io/SnoopCompile.jl/dev/snoopi_deep/#Viewing-the-results). This may help fill in more of that empty space because it captures part of what the compiler is doing.

Unfortunately, when I try to collect inference results using SnoopCompile, I get a stack overflow. And if I increase the stack size, my machine runs out of memory and I cannot produce results.

I can try removing parts of the code to reduce the size of the data structure produced by `@snoopi_deep`.

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 6, 2024, 1:43am UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/58 "2024-07-06T01:43:39Z")

</div>

`julia -t 1`:

 ![perf_t1](https://global.discourse-cdn.com/julialang/original/3X/b/6/b60cfc561d4e439e3b489f11bd1408a8131a0f70.png)  
`julia -O0 -t 1`  
 ![perf_t1_O0](https://global.discourse-cdn.com/julialang/original/3X/5/e/5e979c4146195e0ece0a5868307e2733154d0832.png)

A few observations:

1. What are valleys in the middle?
2. What code is Julia aggressively optimizing? That’s the million dollar question I am seeking
3. In both case, our code (the right-hand-side cluster) takes a relatively short time to compile and execute

---

<div class="post-metadata">

**Author:** ![fatteneder](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fatteneder/32/33991_2.png) [@fatteneder](https://discourse.julialang.org/u/fatteneder)\
**Post date:** [July 6, 2024, 6:23am UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/59 "2024-07-06T06:23:28Z")

</div>

> [@ToucheSir](#):
>
> On earlier (and possibly also recent) versions of Julia, starting Julia with multiple threads and profiling will result in large empty spaces. IIRC the empty space is usually `[width of thread 1 profile] x [N - # of threads]`. Hard to say if that’s the culprit here, but an easy way to check would be to ensure `Threads.nthreads() == 1` before profiling.

Indeed, thanks for the hint.  
I wrongly assumed that each thread gets its separate tab, like it is with newer versions (not sure when that was implemented).

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [July 6, 2024, 9:37am UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/60 "2024-07-06T09:37:08Z")

</div>

> [@osimonn](#):
>
> With regards to system images, we will need to create a system image every time we submit changes to our simulation which can be dozens of times per day. Creating a system image is notoriously slow and it will further exacerbate the job times. Also, for our code, system images will be large and they will be have to be stored for each change, driving up storage cost.

If a significant part of your dependencies rarely change and could be tightly pinned, you could consider making a system image with those dependencies but without your own code.

Actually that’s something I should try out myself at work.

[Previous page](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630.md?page=2)

[Next page](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630.md?page=4)