Potential performance regressions in Julia 1.8 for special un-precompiled type dispatches and how to fix them

TL;DR

With Julia 1.8.0 (as opposed to Julia 1.7.3), for OrdinaryDiffEq v6.24.0 and Trixi.jl v0.4.44, we observe that

  • package installation time increased by 20-50%
  • package loading time increased by 30-50%
  • compilation time increased by 7-10%
  • time-to-first-solution (without precompilation) increased by ~15%
  • runtime of our numerical simulation kernels
    • with bounds checking increased by 7%
    • without bounds checking decreased by 2%

Questions:

  1. Is this a known/expected behavior?
  2. What might be the causes of the observed performance regressions?
  3. Is there something we can (easily) do about it?

Longer version

First of all, kudos and thanks a lot to the JuliaLang developers, maintainers, and contributors, who made Julia 1.8 happen! As one of the maintainers of the Trixi.jl numerical simulation framework, I was very excited about many incoming improvements in terms of speed and usability (Apple Silicon support :partying_face:).

Naturally, we immediately wanted to test how the Trixi.jl performance behaves with Julia 1.8, and we ran some performance tests with a real-world numerical simulation setup, using the current versions of OrdinaryDiffEq (v6.24.0) and Trixi.jl (v0.4.44). To our surprise, we saw that there are number of areas where it seems like Julia 1.8 is actually slower than v1.7. Therefore, I wrote this post to ask for input on what might be the causes of the observed performance regressions, and if there is something we can do to (easily) fix them.

Just to emphasize: I do not want to critize the v1.8 release, I am just curious to find out how we can get our performance results from Julia 1.7 back, and maybe even improve upon them.

Since this is going to be a somewhat lengthy post, I will break it up into several sections.

Setup and measurements

We used the official binaries for Julia 1.7.3 and 1.8.0 on a headless Linux machine. Storage is a fast NVMe SSD. This is the output of versioninfo():

Julia 1.7.3:

Julia Version 1.7.3
Commit 742b9abb4d (2022-05-06 12:58 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: AMD Ryzen Threadripper 3990X 64-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, znver2)

Julia 1.8.0:

Julia Version 1.8.0
Commit 5544a0fab76 (2022-08-17 13:38 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 128 Ɨ AMD Ryzen Threadripper 3990X 64-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, znver2)
  Threads: 1 on 128 virtual cores

For each different performance metric, a separate JULIA_DEPOT_PATH was used to avoid any interference. Each measurement was repeated at least three times, and the results were averaged (with very low mean deviation). All measurements are given in seconds (wall time) and were obtained with serial execution (no threading or MPI etc.).

A collection of the scripts and manifests we used can be found here:

However, please note that this repo is not (yet?) fully self-explanatory :grimacing:

Package installation

Here, we measured how long it takes to use Pkg to install each package into a project.

The increase is 17% for OrdinaryDiffEq and 43% for Trixi.

Package loading

Here, we measured how long it takes to load each package one after the other (OrdinaryDiffEq first, Trixi second). This was done after a full test run, i.e, these times are without precompilation.

The increase is 47% for OrdinaryDiffEq and 33% for Trixi.

Compilation

Here, we measured how long it takes to compile the Julia code (after the packages have already been loaded, i.e., again without precompilation). Compilation time is here computed as the runtime for the first execution of our test setup (a numerical simulation) minus the time for a second execution of the identical code.

The increase in compilation time is 7-10%.

Time-to-first-solution

Here, we measured how long it takes from after Julia has been started until the first simulation is finished (including using times). Again, these times are without precompilation. The results shown are for a run with bounds checking disabled, but the result with bounds checking enabled is very similar.

The increase in the time-to-first-solution is 13-16%.

Runtime of numerical simulation kernels

Here, we show a Trixi-internal performance metric, named ā€œperformance indexā€. It is essentially a measure for the performance of the core algorithms by computing the time of a semidiscretization update per each degree of freedom. Thus again, lower is better/faster.

Here, the story is split: When bounds checking is enabled (using the default), the runtime increases by 7%. When bounds checking is disabled, the runtime decreases by 2%.

Conclusions

It seems like that in our use case, a fairly complex scientific software package for numerical simulation (which has already been performance tuned, e.g., in this paper or this paper), for many of the performance metrics we look at, Julia 1.8 is slower than v1.7. The good news is that the ultimately relevant metric (run time without bounds checking) has improved (if only barely)! However, reiterating my questions from above: It would be great to figure out why this is the case, and what we can do to fix it.

cc @ranocha @jlchan @gregorgassner @tim.holy

15 Likes

The only known effect in Julia v1.8 Iā€™m aware of is that loading time of packages may increase a bit because more code is cached, which should however result in overall slightly faster TTFX. Iā€™ve seen this in a few small packages.

What youā€™re observing instead sounds similar to

Can you see if https://github.com/JuliaLang/julia/pull/46366 helps with loading time? Related issue (also opened by Marius): Large number of invalidations by this package and seems to really slow down certain jll loads Ā· Issue #77 Ā· SciML/Static.jl Ā· GitHub

2 Likes

First of all. Iā€™m writing a whole blog post on this, lots of work to do, etc. Will fill in all details. Short stuff now. But see.

no_jit_lag

Thatā€™s what load times are like now on OrdinaryDiffEq v6.24, so the start there is a bit misleading if someone doesnā€™t know what they are reading. Time to first solution is down an order of magnitude and now it fully precompiles in a system image. Now, I donā€™t think you did that on purpose. You saw this with Trixi.jl, and your statements are true while what I just showed is true. The question is how to reconcile both, and what to do about it.

Iā€™m not going to address the runtime stuff because the 7% is just pure Julia changes, probably inlining and the effects system.

The real thing is time to first solution and package load time. The issue is that everything now fully precompiles on ā€œstandard typesā€ so ā€œmost peopleā€ have much lower load times. Standard types being Vector{Float64} for u0, Float64 for tspan, and NullParameters or Vector{Float64} for parameters. I donā€™t think anyone will disagree that this means almost all users (more than 99%) will experience a faster first solution and all. But what it does mean is that everyone gets those precompiled. Trixi.jl uses its own parameter type, and thus it bypasses this system, and hence you see the increased precompiles time and load time without the benefit.

The answer to this is multi-fold. One,

Using the preferences system to determine what to precompile. For now if you just comment out:

The load times should go back. Please test this. If so, then a preferences system to disable that is the solution.

Two, I have been calling for help to setup upstream SnoopCompiles, so please help. For example, https://github.com/JuliaLinearAlgebra/RecursiveFactorization.jl/blob/v0.2.11/src/RecursiveFactorization.jl that should be changed to snoopprecompile etc. That will reduce the number of repeated compilations and reduce the precompile and load times overall. Every upstream package should probably get a small representative workflow snooped.

Three, see the invalidation report from this week.

The convert overloads is already taken care of, but the Static ones are ongoing.

The big ones are:

Please help with the second one by writing a Cassette pass for LoopVectorization so that it can do function replacement on ! ā†’ static_! so that ! does not need to be overloaded. That would remove most of the recompilation.

Also, it would be helpful to see a representative Trixi invalidation report if you can generate one. Just do exactly as from that DifferentialEquations.jl issue, and share what the top 10 or so invalidators are.

9 Likes

Looking at invalidations, Static.jl with !(::False) is currently the worst when using Trixi.jl. This should hopefully be fixed by Remove invalidating `!` overloads by ChrisRackauckas Ā· Pull Request #78 Ā· SciML/Static.jl Ā· GitHub.

julia> using SnoopCompileCore

julia> invalidations = @snoopr begin
           using Trixi
           trixi_include(default_example())
       end

julia> trees = invalidation_trees(invalidations);

Edit: But also see the list of PRs fixing related invalidations in the post below.

1 Like

Okay, so I went a bit invalidation hunting this morning:

However, I would have expected that these invalidations happen also with Julia v1.7 - or do I miss something?

14 Likes

Thereā€™s two facts that collide to make this matter more now. One is:

This means that if a precompile is missing in package X that is used/needed to precompile a call in package Y, it will now precompile with the ownership of package Y. This has 2 effects: one is that more precompilation will happen, two is that if package Z also needs the missing precompile, then package Y and package Z will precompile their own versions of the call from package X.

The more precompilation will increase load times but normally decrease first solve time, if types tend to match etc. But, it will increase load times more if a call is precompiled multiple times. The solution then is to try and precompile ā€œwhat we know is neededā€ in package X, and use the system of external precompilation as sparingly as possible. Itā€™s required to make things work (for example, Base misses precompilation of Vector(::Uninitiaialized,::Tuple) so oops you might need that), but donā€™t overrely on it.

The next is how SnoopPrecompile changes the game:

The main fact is that uninferred calls can now do precompilation effectively. Go back and re-read this issue in full:

That was the old situation. The issue was, if we can get inference to happen higher, then precompilation will happen on RecursiveFactorization.jl and that will send the first solve time with implicit methods from 22 seconds to 3. Now with SnoopPrecompile, the pre-changed version probably already hits 3 (on current release, itā€™s now 0.5 seconds BTW).

Basically, this means a lot more precompiles. But because a lot more precompiles, doubling precompiles hurts more. And invalidating functions hurts even more. If you do @time using OrdinaryDiffEq, the real important stat is 75% of the time is recompilation. This is invalidations taking what was precompiled and throwing it away because loading a different package (Static.jl, LoopVectorization.jl) invalidates the precompiled version.

So in the end, a lot more gets precompiled so the load time is increased (because of the ownership and non-inferred help), this does have a major improvement on the first solve time, but it increases using time, which then explodes because invalidations throw away more than a majority of that precompile work.

Therefore, invalidations matter a whole lot more now. Itā€™s time to fix as much as we can there.

3 Likes

Yeah, right, thatā€™s explains (at least a part of) this.

Out of curiosity: Are invalidation fixes usually backported (to release-1.8 in this case) or do we have to wait for Julia v1.9?

But do we have tools that can be used for example in CI to make sure invalidations arenā€™t brought back again in the future? The fact is that waiting for someone enough pissed off to hunt down all the invalidations can work once, but isnā€™t much sustainable in the long run.

13 Likes

I think at a package level, one may assert that a PR doesnā€™t add invalidations. See e.g. https://github.com/JuliaArrays/OffsetArrays.jl/blob/master/.github/workflows/invalidations.yml

8 Likes

First of all, thanks to everyone who offered some helpful suggestions!

Unfortunately, no. When I try to do using Trixi with the nightly build 36aab14a97, Julia segfaults with

[1693377] signal (11): Segmentation fault
in expression starting at REPL[1]:1
ijl_array_del_end at /cache/build/default-amdci5-3/julialang/julia-master/src/array.c:1144
jl_insert_method_instances at /cache/build/default-amdci5-3/julialang/julia-master/src/dump.c:2379 [inlined]
_jl_restore_incremental at /cache/build/default-amdci5-3/julialang/julia-master/src/dump.c:3273
[...]
Full error message
[1693377] signal (11): Segmentation fault
in expression starting at REPL[1]:1
ijl_array_del_end at /cache/build/default-amdci5-3/julialang/julia-master/src/array.c:1144
jl_insert_method_instances at /cache/build/default-amdci5-3/julialang/julia-master/src/dump.c:2379 [inlined]
_jl_restore_incremental at /cache/build/default-amdci5-3/julialang/julia-master/src/dump.c:3273
ijl_restore_incremental at /cache/build/default-amdci5-3/julialang/julia-master/src/dump.c:3333
_include_from_serialized at ./loading.jl:867
_require_search_from_serialized at ./loading.jl:1099
_require at ./loading.jl:1378
_require_prelocked at ./loading.jl:1260
macro expansion at ./loading.jl:1240 [inlined]
macro expansion at ./lock.jl:267 [inlined]
require at ./loading.jl:1204
jfptr_require_50250.clone_1 at /mnt/hd1/opt/julia/nightly-20220825-36aab14a97/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-master/src/gf.c:2447 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-master/src/gf.c:2629
jl_apply at /cache/build/default-amdci5-3/julialang/julia-master/src/julia.h:1854 [inlined]
call_require at /cache/build/default-amdci5-3/julialang/julia-master/src/toplevel.c:466 [inlined]
eval_import_path at /cache/build/default-amdci5-3/julialang/julia-master/src/toplevel.c:503
jl_toplevel_eval_flex at /cache/build/default-amdci5-3/julialang/julia-master/src/toplevel.c:731
eval_body at /cache/build/default-amdci5-3/julialang/julia-master/src/interpreter.c:561
eval_body at /cache/build/default-amdci5-3/julialang/julia-master/src/interpreter.c:522
jl_interpret_toplevel_thunk at /cache/build/default-amdci5-3/julialang/julia-master/src/interpreter.c:751
jl_toplevel_eval_flex at /cache/build/default-amdci5-3/julialang/julia-master/src/toplevel.c:912
jl_toplevel_eval_flex at /cache/build/default-amdci5-3/julialang/julia-master/src/toplevel.c:856
ijl_toplevel_eval_in at /cache/build/default-amdci5-3/julialang/julia-master/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
eval_user_input at /cache/build/default-amdci5-3/julialang/julia-master/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:152
repl_backend_loop at /cache/build/default-amdci5-3/julialang/julia-master/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:248
#start_repl_backend#46 at /cache/build/default-amdci5-3/julialang/julia-master/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:233
start_repl_backend##kw at /cache/build/default-amdci5-3/julialang/julia-master/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:230 [inlined]
#run_repl#59 at /cache/build/default-amdci5-3/julialang/julia-master/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:372
run_repl at /cache/build/default-amdci5-3/julialang/julia-master/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:357
jfptr_run_repl_57495.clone_1 at /mnt/hd1/opt/julia/nightly-20220825-36aab14a97/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-master/src/gf.c:2447 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-master/src/gf.c:2629
#1007 at ./client.jl:413
jfptr_YY.1007_36884.clone_1 at /mnt/hd1/opt/julia/nightly-20220825-36aab14a97/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-master/src/gf.c:2447 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-master/src/gf.c:2629
jl_apply at /cache/build/default-amdci5-3/julialang/julia-master/src/julia.h:1854 [inlined]
jl_f__call_latest at /cache/build/default-amdci5-3/julialang/julia-master/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:810 [inlined]
invokelatest at ./essentials.jl:807 [inlined]
run_main_repl at ./client.jl:397
exec_options at ./client.jl:314
_start at ./client.jl:514
jfptr__start_30331.clone_1 at /mnt/hd1/opt/julia/nightly-20220825-36aab14a97/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-3/julialang/julia-master/src/gf.c:2447 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-3/julialang/julia-master/src/gf.c:2629
jl_apply at /cache/build/default-amdci5-3/julialang/julia-master/src/julia.h:1854 [inlined]
true_main at /cache/build/default-amdci5-3/julialang/julia-master/src/jlapi.c:567
jl_repl_entrypoint at /cache/build/default-amdci5-3/julialang/julia-master/src/jlapi.c:711
main at julia-nightly-20220825-36aab14a97 (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
Allocations: 22294390 (Pool: 22283733; Big: 10657); GC: 12
Segmentation fault (core dumped)

Just to re-emphasize this from my original post: I do not mean to criticize any particular package.

However, at the moment your description does not match our observations from a (what I would call) regular userā€™s perspective: If we install OrdinaryDiffEq and Trixi into a fresh depot on a standard Linux machine, then the package installation time and the package loading time and the compilation time go up from 1.7.3 to 1.8.0.

These times do not just represent ā€œconvenienceā€ issues for us: Longer package installation times means increased development times due to higher CI wait times. Longer loading times means it is harder to use these packages for quick demonstrations or for live experimentation when teaching university courses. Longer compilation times are problematic when running parallel jobs on supercomputers.

Actually, this is IMHO probably the biggest issue of all: I would assume that a language designed for high performance will - unless specifically announced - only ever have improved execution performance with each new release. Thus, these measurements were at least a surprise to us. I would be very interested in hearing from others if this regression has been observed for other use cases as well. Since we use Julia as an HPC language, a 7% regression in execution speed is non-negligible.

They improve, but unfortunately not completely back to Juila 1.7.3 levels:

P.S.: It seems like the title of my post was unilaterally changed by someone to something different from what I wrote. I think this is somewhat rude, especially since now the title does not fully reflect my original intent anymore (in my opinion, the performance regressions are only a question of type dispatches, given that compilation and execution performance is affected as well).

4 Likes

Note that `Core.ifelse` calls to avoid invalidations from defining custom `Base.ifelse` methods by chriselrod Ā· Pull Request #46366 Ā· JuliaLang/julia Ā· GitHub isnā€™t merged, so youā€™d have to compile julia yourself (maybe applying that patch on top of v1.8.0 tag, to minimise unrelated changes).

Youā€™re missing the huge caveat and the whole point. Your statement only for cases like Trixi.jl where special parameters types are used. Thatā€™s a huge deal. The first solve time is dramatically lower if thatā€™s not the case. Thatā€™s pretty clear from the measurements. Regular users use Vector{Float64} or nothing for parameters: if you donā€™t believe thatā€™s the case please provide evidence (I can tell you that from thousands of Discourse posts, more than 99% of them are in this case!). Yes, for the cases that people post about <1% of the time, which happens to be the case that you are looking at all of the time, there is this problem. I understand that increased Trixi compilation time, plus v1.8 changes and increased invalidations, but there is no reason to go doom and gloom beyond whatā€™s actually true. Recognition of this fact is what will lead us to the real issues and the real solutions.

That is convenience. Iā€™m sorry itā€™s now less convenient, but we will fix this. I need your help though on categorizing and profiling your case though in order to do this efficiently. My compute resources are swamped trying to survey the possible cases.

Note that you can cache the precompilation (or do a system image build thatā€™s cached) that would remove this, so for CI infrastructure there are some easy fixes. If you need it, we can also get you some more CI compute on the AMDCI machines. In fact, Iā€™m curious whether @giordano has any build scripts that do a single precompilation step for a multi-group CI test.

Yes, but thatā€™s a completely different thread. Please create a separate thread on this. Itā€™s different profiles, different causes and effects, etc. It would just be confusing to address it here because it has nothing to do with the compile times which is a whole discussion of its own. As I said, I think itā€™s due to the effects system, we can take a look at a thread with profiles and everything, but putting all of that into a thread about precompilation would be unreadable so itā€™s best to keep two completely separate topics in separate threads. Iā€™d be happy to dig into this with you, but thatā€™s 25+ posts with images etc. on its own. Handling this precompilation is already long.

The CI builds have prebuilt artifacts you can download. I just learned that the other day: itā€™s so much easier :sweat_smile:.

I did that and Iā€™ll take full responsibility. I donā€™t think itā€™s rude because if someone finds this thread they will find 13 in-depth posts about v1.8 precompilation changes and how it adversely effects the special type cases which are not covered by the package snooping. They will find nothing about runtime changes in v1.8, which is a completely separate topic. Keeping things organized and searchable is helpful. But again, thereā€™s no reason to not discuss v1.8 runtime changes, itā€™s just a separate thread and a tangent in a discussion about precompilation.

3 Likes

Generally these arenā€™t cases that just come and go. These Static, LoopVectorization, Symbolic, and ChainRules core invalidations have been there for a long time. Itā€™s just that precompilation never really did much, so no one really cared.

It would be good to add invalidation testing to infrastructure somehow, but solving the root cases in the core packages that cause the vast majority of issues is relatively maintainable. There just arenā€™t that many packages that are used by the majority of Julia users and which happen to overload something like !.

1 Like

Can you share the recompilation percentage on v1.8 that youā€™re seeing? @time on the using should show that.

Also share @time_imports Trixi

Sure, ā€œbrought backā€ wasnā€™t the best word choice, I really meant how to not introduce new one in the future. It looks like the script suggested by Jishnu above can help with that.

Oh I see what you were asking for now. Yeah, the script can work, but itā€™s overly sensitive. ā€œMostā€ invalidations donā€™t mean very much, so getting a red test from adding one invalidation to a high level call is a bit too conservative. You almost want to pair it with a cost model of precompilation. We might adopt it in SciML if someone would help us slam the script around 100 repos, though there would need to be some judgement calls made it on (the output would help make said judgements though!)

To keep people up to date: We distributed an updated variant of the script mentioned above to all SciML repos and the Trixi framework. I also made quite a few PRs to other basic packages in the ecosystem. Letā€™s see how things evolve from here on and letā€™s work together to fix invalidations!

8 Likes

And there was a major change to the function wrapping for late wrapping that should help first solve times for all downstream users, even Trixi, if Trixi snoop compiles. Thatā€™ll get a write-up soon. Also, other changes like A bunch of ambiguity fixes by ChrisRackauckas Ā· Pull Request #1753 Ā· SciML/OrdinaryDiffEq.jl Ā· GitHub

2 Likes