Precompiled native code seems unused when a package is sourced from a git rev instead of a local path

General Usage
, ,
1

On Julia 1.12.6, the same package at the same commit runs ~20x slower when it comes from Pkg.add(url=..., rev=...) than from a local path. The hot workload is a ModelingToolkit-generated, RuntimeGeneratedFunctions-wrapped ODE right-hand side that the package compiles in a PrecompileTools.@compile_workload; with the git-rev source it appears to be re-JIT’ed in every fresh process.

Repro (V3Kite.jl v1.0.1, ~3 min precompile per environment, two fresh processes):


using Pkg; Pkg.activate(tempname())

Pkg.add(url="https://github.com/OpenSourceAWE/V3Kite.jl", rev="v1.0.1")
Pkg.precompile()
using V3Kite; @time init(9.0, 200.0)      # 38.2 s

Fresh process; local clone of the same repo, checked out at v1.0.1

using Pkg; Pkg.activate(tempname())

Pkg.develop(path="/path/to/V3Kite.jl")
Pkg.precompile()
using V3Kite; @time init(9.0, 200.0)      # 1.8 s
V3Kite source plain julia --project with a PackageCompiler sysimage
url + rev 38.2 s 33.6 s
path 1.77 s 2.25 s

Under the git-rev source the workload does run: Pkg.precompile() logs it executing init, and leaves a ~141 MB pkgimage — same size class as the path build. So it compiles and is cached to disk, the native code is simply not used at runtime. Ruled out: the system image, the package’s own on-disk model cache, and the RuntimeGeneratedFunctions version (identical git-tree-sha1 in both manifests). Full measurement log incl. the dead ends: docs/sysimage_notes.md.

Is this expected — does a git-rev source invalidate RGF-backed precompiled code somehow — or is it a bug?

Disclaimer:
Written by Claude, manually proof-read and corrected

This is somewhat annoying, because it means I can make the original V3Kite package fast, but if I use that package as a dependency in another package it will be slow. Unless I dev it.

2

I have seen cases where RGF code does not get cached but that has been when it is put into a sysimage and even that case should have been fixed.

Tell the robot to run with --trace-compile and see what signatures are getting compiled and ask it to try figure out the root cause.

3

One more aspect of the issue:

We serialize the simplified model in a bin file. A bin file created from our program without a system image cannot be read by the same program with a system image, and vice versa.

I get the error message:

Deserialization failed at 1.785854577145231e9
Path: /home/ufechner/.julia/scratchspaces/4caac9c8-c726-438f-ab10-3553e918eab1/v3kite_cache/model_v0.11.1_jl1.12_v3_particle_dir_dynamic_44pnt_95seg_0grp_1wng_1wch.bin
ArgumentError: cannot deserialize a dropped RuntimeGeneratedFunction; serialize it before calling drop_expr
Stacktrace:
  [1] deserialize(s::Serialization.AbstractSerializer, ::Type{RuntimeGeneratedFunctions.RuntimeGeneratedFunction{(:___mtkunknowns___, :___mtkparameters___, :__argₛᵧₘ1249670312406203976), ModelingToolkitBase.var"#_RGF_ModTag", ModelingToolkitBase.var"#_RGF_ModTag", (0xcfd89363, 0x1321e11e, 0x550d3a0b, 0x3787cb31, 0xc00bf369), Nothing}})
    @ RuntimeGeneratedFunctions ~/.julia/packages/RuntimeGeneratedFunctions/odmYb/src/RuntimeGeneratedFunctions.jl:571
  [2] handle_deserialize(s::Serialization.Serializer{IOStream}, b::Int32)
    @ Serialization ~/.julia/juliaup/julia-1.12.6+0.x64.linux.gnu/share/julia/stdlib/v1.12/Serialization/src/Serialization.jl:915
  [3] deserialize(s::Serialization.Serializer{IOStream}, t::DataType)

The full error log can be found here.

I already see one potential issue: there might be multiple versions of the bin file in different locations. The file created during pre-compilation is going to a scratchspace, bin files created during normal operation not.

UPDATE:
There are four evironments being used, the main and the examples environment of SimpleKiteControllers, and the main and the examples environment of V3Kite. The main environment of V3Kite still had the broken version of RuntimeGeneratedFunctions. That is fixed now.

5

45 s of re-JIT because precompilation compiled against a different serialized model

We had an ODE init that cost 45 s in one environment and 2.5 s in another, with identical code, identical Manifest, identical system image. The only difference was whether our plant package was sourced by {url, rev} or by {path}, which made no sense.

It had nothing to do with [sources]. The model is a ModelingToolkit system serialized to a .bin, so its RHS is a RuntimeGeneratedFunction — and the RGF id (a hash of the generated body) is a type parameter:

ODEFunction{true, AutoSpecialize, GeneratedFunctionWrapper{(2,3,true),
    RuntimeGeneratedFunction{(...), _RGF_ModTag, _RGF_ModTag, (0xc03b7c9f, 0x507d5098, ...)}}}

The package’s @compile_workload runs init, which caches the external specializations keyed to that type into its pkgimage — DiffEqBase.promote_f (22 s to compile!), 74 SymbolicIndexingInterface observed-getter closures, the RGF generated_callfunc’s.

But two independently built model binaries get different RGF ids. Ours were:

  • data/*.bin is gitignored, so a Pkg-installed copy ships no model — the workload builds its own into scratchspaces/<uuid>/;
  • a deved checkout instead caches into its own writable data/, which is the file our downstream project had copied.

So under url/rev, every specialization in the pkgimage was keyed to a type that never came into existence at runtime, and all of it re-JIT’ed. Swapping only the .bin gives a clean crossover — the source contributes nothing:

model binary used at runtime path source url/rev source
built by the dev checkout 6.2 s 47.1 s
built by the workload in the scratchspace 47.1 s 5.7 s

Fix: stop overriding cache_path, so the run deserializes exactly the binary the workload compiled against. 44.8 s → 2.9 s, no sysimage, no dev needed.

Lesson: if your precompile workload deserializes generated code, precompilation only pays off when the runtime loads the same bytes. Ship the artifact, or make sure the workload and the run resolve to one cache — otherwise @compile_workload is burning 40 s per precompile on code nobody will ever use.