How to debug incomplete precompilation

Torkel · September 9, 2025, 9:33am

I am having a problem that more or less every time I start Julia I run into precompilations that take ages (I can literally have hours each day where Julia is unavailable due to this).

When I do using MyPackage I get a lot of stuff like this:

  ? NonlinearSolve
  ? OrdinaryDiffEqNonlinearSolve
  ? ModelingToolkit
  ? OrdinaryDiffEqPDIRK
  ? OrdinaryDiffEqStabilizedIRK
  ? OrdinaryDiffEqSDIRK
  ? OrdinaryDiffEqIMEXMultistep
  ? OrdinaryDiffEqFIRK
  ? ModelingToolkit → MTKChainRulesCoreExt
  ? OptimizationBase → OptimizationMTKExt
  ? ModelingToolkitStandardLibrary

Which, according to this, seems to indicate:

? A question mark character indicates that a PrecompilableError was thrown, indicating that precompilation was disallowed, i.e. __precompile__(false) in that package.
(Here I am in a folder MyFolder, which contains a package-folder Mypackage, which I have added using ]dev MyPackage)

This seems to possibly make sense? I.e. all these packages have some PrecompilationError, meaning that the precompilation fails, which could (maybe?) both explain that it takes lots of time and that I have to keep rerunning precompilation even though I have not updated any package versions.

But how do I actually figure out what is going on? Many of these are fairly normal packages, and it seems unlikely that all of these (e.g. NonlinearSolve, OrdinaryDiffEq, Optimization) all have fundamental problems preventing precompilations. I have searched the packages for __precompile__(false) statements, but there does not seem to be any.

At the end of precompilation, I get a lot of messages like

┌ LinearSolve → LinearSolveSparseArraysExt
│  ┌ Warning: Module RecursiveArrayTools with build ID fafbfcfd-0513-4542-0020-32fcd9b97709 is missing from the cache.
│  │ This may mean RecursiveArrayTools [731186ca-8d62-57ce-b412-fbd966d074cd] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:2541
│  ┌ Error: Error during loading of extension RecursiveArrayToolsSparseArraysExt of RecursiveArrayTools, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Error when precompiling module, potentially caused by a __precompile__(false) declaration in the module.
│  │    Stacktrace:
│  │      [1] _require(pkg::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:2603
│  │      [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:2388
│  │      [3] #invoke_in_world#3
│  │        @ ./essentials.jl:1089 [inlined]
│  │      [4] invoke_in_world
│  │        @ ./essentials.jl:1086 [inlined]
│  │      [5] _require_prelocked
│  │        @ ./loading.jl:2375 [inlined]
│  │      [6] _require_prelocked
│  │        @ ./loading.jl:2374 [inlined]
│  │      [7] run_extension_callbacks(extid::Base.ExtensionId)
│  │        @ Base ./loading.jl:1544
│  │      [8] run_extension_callbacks(pkgid::Base.PkgId)
│  │        @ Base ./loading.jl:1576
│  │      [9] run_package_callbacks(modkey::Base.PkgId)
│  │        @ Base ./loading.jl:1396
│  │     [10] _tryrequire_from_serialized(modkey::Base.PkgId, build_id::UInt128)
│  │        @ Base ./loading.jl:1890
│  │     [11] _tryrequire_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String)
│  │        @ Base ./loading.jl:1956
│  │     [12] _require(pkg::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:2594
│  │     [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:2388
│  │     [14] #invoke_in_world#3
│  │        @ ./essentials.jl:1089 [inlined]
│  │     [15] invoke_in_world
│  │        @ ./essentials.jl:1086 [inlined]
│  │     [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:2375
│  │     [17] macro expansion
│  │        @ ./loading.jl:2314 [inlined]
│  │     [18] macro expansion
│  │        @ ./lock.jl:273 [inlined]
│  │     [19] __require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:2271
│  │     [20] #invoke_in_world#3
│  │        @ ./essentials.jl:1089 [inlined]
│  │     [21] invoke_in_world
│  │        @ ./essentials.jl:1086 [inlined]
│  │     [22] require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:2260
│  │     [23] include
│  │        @ ./Base.jl:557 [inlined]
│  │     [24] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│  │        @ Base ./loading.jl:2881
│  │     [25] top-level scope
│  │        @ stdin:6
│  │     [26] eval
│  │        @ ./boot.jl:430 [inlined]
│  │     [27] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│  │        @ Base ./loading.jl:2734
│  │     [28] include_string
│  │        @ ./loading.jl:2744 [inlined]
│  │     [29] exec_options(opts::Base.JLOptions)
│  │        @ Base ./client.jl:321
│  │     [30] _start()
│  │        @ Base ./client.jl:531
│  └ @ Base loading.jl:1550
│  ┌ Warning: Module SciMLBase with build ID fafbfcfd-f59a-ce42-0020-32f7a6c767b1 is missing from the cache.
│  │ This may mean SciMLBase [0bca4576-84f4-4d90-8ffe-ffa030f20462] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:2541
│  ┌ Error: Error during loading of extension SciMLBaseChainRulesCoreExt of SciMLBase, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Error when precompiling module, potentially caused by a __precompile__(false) declaration in the module.
│  │    Stacktrace:
│  │      [1] _require(pkg::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:2603
│  │      [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:2388
│  │      [3] #invoke_in_world#3
│  │        @ ./essentials.jl:1089 [inlined]
│  │      [4] invoke_in_world
│  │        @ ./essentials.jl:1086 [inlined]
│  │      [5] _require_prelocked
│  │        @ ./loading.jl:2375 [inlined]
│  │      [6] _require_prelocked
│  │        @ ./loading.jl:2374 [inlined]
│  │      [7] run_extension_callbacks(extid::Base.ExtensionId)
│  │        @ Base ./loading.jl:1544
│  │      [8] run_extension_callbacks(pkgid::Base.PkgId)
│  │        @ Base ./loading.jl:1576
│  │      [9] run_package_callbacks(modkey::Base.PkgId)
│  │        @ Base ./loading.jl:1396
│  │     [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:2399
│  │     [11] #invoke_in_world#3
│  │        @ ./essentials.jl:1089 [inlined]
│  │     [12] invoke_in_world
│  │        @ ./essentials.jl:1086 [inlined]
│  │     [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:2375
│  │     [14] macro expansion
│  │        @ ./loading.jl:2314 [inlined]
│  │     [15] macro expansion
│  │        @ ./lock.jl:273 [inlined]
│  │     [16] __require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:2271
│  │     [17] #invoke_in_world#3
│  │        @ ./essentials.jl:1089 [inlined]
│  │     [18] invoke_in_world
│  │        @ ./essentials.jl:1086 [inlined]
│  │     [19] require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:2260
│  │     [20] include
│  │        @ ./Base.jl:557 [inlined]
│  │     [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│  │        @ Base ./loading.jl:2881
│  │     [22] top-level scope
│  │        @ stdin:6
│  │     [23] eval
│  │        @ ./boot.jl:430 [inlined]
│  │     [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│  │        @ Base ./loading.jl:2734
│  │     [25] include_string
│  │        @ ./loading.jl:2744 [inlined]
│  │     [26] exec_options(opts::Base.JLOptions)
│  │        @ Base ./client.jl:321
│  │     [27] _start()
│  │        @ Base ./client.jl:531
│  └ @ Base loading.jl:1550
│  ┌ Warning: Module JLLWrappers with build ID fafbfcfd-dacd-7e66-0020-30fb8e2fed08 is missing from the cache.
│  │ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:2541
└  
┌ OptimizationBase → OptimizationMTKExt
│  ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for SciMLBase [0bca4576-84f4-4d90-8ffe-ffa030f20462]
│  │   exception = Required dependency Base.PkgId(Base.UUID("3cdcf5f2-1ef4-517c-9805-6587b60abb01"), "RecipesBase") failed to load from a cache file.
│  └ @ Base loading.jl:2596
└

┌ Latexify → DataFramesExt
│  ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PrettyTables [08abe8d2-0d0c-5749-adfa-8a2ac140af0d]
│  │   exception = Required dependency Base.PkgId(Base.UUID("bd369af6-aec1-5ad0-b16a-f7cc5008161c"), "Tables") failed to load from a cache file.
│  └ @ Base loading.jl:2596
└  
┌ OptimizationOptimisers
│  ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for PrettyTables [08abe8d2-0d0c-5749-adfa-8a2ac140af0d]
│  │   exception = Required dependency Base.PkgId(Base.UUID("bd369af6-aec1-5ad0-b16a-f7cc5008161c"), "Tables") failed to load from a cache file.
│  └ @ Base loading.jl:2596
└  
┌ OrdinaryDiffEqRKN
│  ┌ Warning: The call to compilecache failed to create a usable precompiled cache file for RecursiveArrayTools [731186ca-8d62-57ce-b412-fbd966d074cd]
│  │   exception = Required dependency Base.PkgId(Base.UUID("3cdcf5f2-1ef4-517c-9805-6587b60abb01"), "RecipesBase") failed to load from a cache file.
│  └ @ Base loading.jl:2596
└  
┌ NonlinearSolveBase → NonlinearSolveBaseLineSearchExt
│  ┌ Warning: Module RecipesBase with build ID fafbfcfd-0d4e-ed10-0020-311511a516d1 is missing from the cache.
│  │ This may mean RecipesBase [3cdcf5f2-1ef4-517c-9805-6587b60abb01] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:2541
└  
┌ NonlinearSolveSpectralMethods → NonlinearSolveSpectralMethodsForwardDiffExt
│  ┌ Warning: Module JLLWrappers with build ID fafbfcfd-dacd-7e66-0020-30fb8e2fed08 is missing from the cache.
│  │ This may mean JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:2541
└

which might provide some hint. However, checking e.g. GitHub - SciML/RecursiveArrayTools.jl: Tools for easily handling objects like arrays of arrays and deeper nestings in scientific machine learning (SciML) and other applications, there is no mention of __precompile__(false).

Generally, would it be possible to a few extensions of some packages like this prevents precompilation for the entire system?

ChatGPT says that unless there are ✗ PackageName in the precompilation output I should not worry, and things does run properly. However, it is getting increasingly unsistaianble with these really long precompilation times, and I feel I need to figure out what is actually going on.

ufechner7 · September 9, 2025, 9:56am

Can you share the output of

julia> using Pkg
julia> Pkg.status()

?

And perhaps also the output of:

versioninfo()

?

Torkel · September 9, 2025, 12:56pm

Sorry for the delay (took a while for everything to finish precompiling).

julia> using Pkg

julia> Pkg.status()
Status `~/Desktop/MyProject/Project.toml`
  [336ed68f] CSV v0.10.15
  [479239e8] Catalyst v15.0.5 `/home/loman/.julia/dev/Catalyst#my_branch`
  [d3d80556] LineSearches v7.4.0
⌃ [b2108857] Lux v1.13.0
⌃ [f162e290] ModelingToolkitNeuralNets v1.7.0
  [9af52f01] MyPackage v0.1.0 `MyPackage`
⌃ [7f7a1694] Optimization v4.3.0
⌃ [91a5bcdd] Plots v1.40.13
  [860ef19b] StableRNGs v1.0.3
  [f3b207a7] StatsPlots v0.15.7
Info Packages marked with ⌃ have new versions available and may be upgradable.

julia> versioninfo()
Julia Version 1.11.5
Commit 760b2e5b739 (2025-04-14 06:53 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 96 × Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, icelake-server)
Threads: 4 default, 0 interactive, 2 GC (on 96 virtual cores)
Environment:
  JULIA_EDITOR = code
  JULIA_VSCODE_REPL = 1
  JULIA_NUM_THREADS = 4

ufechner7 · September 9, 2025, 1:05pm

Did you try:

using Pkg
Pkg.update()

?
And can you share the Projects.toml file of MyPackage?

And is there a specific reason that you are on Catalyst#my_branch of Catalyst?

Regarding your original question: If you have precompilation errors during package development, then

create a project where you only add released packages
update this project to the newest versions with Pkg.update()
If precompilation now fails, report here on Discourse and ask for help.
if this works, switch to the custom branch of Catalist and see if precompilation still works
If it still works, the problem is not in your custom branch. If it fails, compare your custom branch and the main branch of Catalist.jl , in particular the Projects.toml file. Perhaps you need to rebase your custom branch if it got outdated?
Add your new package MyPackage. If precompilation now fails you know that MyPackage is the reason. Most likely there is either something wrong with the Project.toml of MyPackage, like too restrictive limits in the [compat] section. Or MyPackage depends on a problematic package, for example a package that has not been maintained for a long time.

Torkel · September 9, 2025, 2:17pm

I was afraid of doing

using Pkg
Pkg.update()

since it might update some package and things don’t work anymore (yes, I don’t fully trust people to fully follow semantic versioning and rather not take any risks).

I have to use Catalyst#my_branch as there is some (very slight) customisation that I have needed to do that is not avaibale in the main package.

This is the Project.toml file

name = "MyPackage"
uuid = "9af52f01-499f-42ca-a116-377b7f889deb"
authors = ["Torkel Loman <torkel.loman@gmail.com>"]
version = "0.1.0"

[deps]
CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
Catalyst = "479239e8-5488-4da2-87a7-35f2df7eef83"
ComponentArrays = "b0b7db55-cfe3-40fc-9ded-d10e2dbeff66"
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
KernelDensity = "5ab0869b-81aa-558d-bb23-cbf5423bbe9b"
LikelihoodProfiler = "93acb638-a083-5915-8dce-d129bc6a3f59"
LineSearches = "d3d80556-e9d4-5f37-9878-2ab0fcc64255"
Lux = "b2108857-7c20-44ae-9111-449ecde12c47"
ModelingToolkit = "961ee093-0014-501f-94e3-6117800e7a78"
ModelingToolkitNeuralNets = "f162e290-f571-43a6-83d9-22ecc16da15f"
Optimization = "7f7a1694-90dd-40f0-9382-eb1efda571ba"
OptimizationBBO = "3e6eede4-6085-4f62-9a71-46d9bc1eb92b"
OptimizationNLopt = "4e6fcdb7-1186-4e1f-a706-475e75c168bb"
OptimizationOptimJL = "36348300-93cb-4f02-beb5-3c3902f8871e"
OptimizationOptimisers = "42dfb2eb-d2b4-4451-abcd-913932933ac1"
OrdinaryDiffEq = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed"
Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
QuasiMonteCarlo = "8a4e6c94-4038-4cdc-81c3-7e6ffdb2a71b"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
Serialization = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
StableRNGs = "860ef19b-820b-49d6-a774-d7a799459cd3"

[compat]
CSV = "0.10.15"
Catalyst = "15.0.2"
ComponentArrays = "0.15.26"
Distributions = "0.25.118"
KernelDensity = "0.6.9"
LikelihoodProfiler = "1.0.1"
LineSearches = "7.4.0"
Lux = "1.10.0"
ModelingToolkit = "9.73.0"
ModelingToolkitNeuralNets = "1.7.0"
Optimization = "4.1.2"
OptimizationBBO = "0.4.0"
OptimizationNLopt = "0.3.2"
OptimizationOptimJL = "0.4.3"
OptimizationOptimisers = "0.3.7"
OrdinaryDiffEq = "6.93.0"
Plots = "1.40.11"
QuasiMonteCarlo = "0.3.3"
Random = "1.11.0"
Serialization = "1.11.0"
StableRNGs = "1.0.2"

Thanks for the step-by-step list as well, I will do that. Seems like there is some fundamental issue at least.

ufechner7 · September 9, 2025, 2:49pm

This is the wrong approach. If you are afraid that things break, making a backup of your Manifest.toml should be sufficient.

Additionally you can add a compat section to your Project.toml file where each of the packages gets the current version assigned. Or you use the freeze(Pkg) function of PkgHelpers.jl that does that for you.

But the point is, you should - during development - always update to the latest version of all packages, and only downgrade if something brakes. To know if anything brakes you need unit tests. And you can downgrade just by setting an upper limit for the version of a package in the [compat] section.

Torkel · September 9, 2025, 2:54pm

Got it,

just in case I misunderstand something here; is there a qualitative difference to freeze(Pkg) my environment to just not updating it? Or does freeze(Pkg) technically permit some updates to downstreams packages which just not updating would dissalow?

Jake · September 9, 2025, 3:21pm

PackageCompatUI.jl is very useful for setting the Compat bounds of packages, if you need to set them.

Benny · September 9, 2025, 4:09pm

That’s one possible reason but not the only one. I don’t know of a list of reasons and how to address them, wish I did.

That might be a fair approach if everything was working and you need reproducibility far more than updates, but precompilation is already failing and many patches exist exactly to fix that.

Yep.

ufechner7 · September 9, 2025, 4:25pm

Yes. If you call freeze, you only disable updates to packages in Project.toml, but not to indirect dependencies.

Furthermore, you should also not freeze all packages. I would use this command only as starting point to get the required compat entries into the project file, then relax the rules one by one as long as the tests pass.

Relaxing means going from ControlPlots = "=0.2.9" to ControlPlots = "~0.2.9" to ControlPlots = "0.2.9", just as example.

And yes, this is a bit of a pain. If you have complex packages you need a fast computer and spend ~10% of your development time on maintenance of Package.toml. But that is the price you pay for a. having super up-to-date software packages available and b. not having to write 95% of the code yourself.

Torkel · September 9, 2025, 4:32pm

Ok, thanks everyone.

It sounds like the recommendation is to look for a specific package which might cause the problem, and then file a corresponding report? I can do that, that makes sense.

(my initial thought was that I trusted the packages, and that there rather might be some form of system-level thing which cases packages to not be precompiled properly)

ufechner7 · September 9, 2025, 4:34pm

Not exactly. Often a curtain combination of packages causes these problems, mainly if they are outdated. So if you share the output of Pkg.status() here, then people can help you to identify and fix these problems. Sometimes also the content of your .julia folder might be corrupted. Then deleting it might help (but make a backup first).

We have many, many degrees of freedom with Julia (combining different packages on different operating systems with different Julia versions), so there is no “one-size-fits-all” solution.

If you have a well-tested piece of code, always create a backup of your Manifest.toml file and push it to git. I usually rename it to Manifest.toml.default or Manifest-v1.11.toml.default before I add it to git.

Still worth reading: Working with Julia projects | Julia programming notes

Torkel · September 9, 2025, 4:43pm

Thanks, that is very useful.

Given that I have some custom code, it seems however a first step is to ioslate the issue as something which does not depend on MyPackage and my special branch of Catalyst.

Thanks for the link as well, seems I have some reading and contemplation to do.

Topic		Replies	Views
Precompilation takes ages General Usage	14	1373	January 5, 2024
Why is this dependency not precompiling? New to Julia	15	3452	December 13, 2021
Precompile package causing missing-from-cache errors New to Julia precompilation	0	1114	March 11, 2019
Precompilation interrupted (bug?) General Usage package , precompilation	2	136	June 20, 2024
Precompiling error for unknown reason New to Julia	10	613	December 3, 2023

How to debug incomplete precompilation

Related topics