Why is __init__() for a package called during precompilation of package extensions? I thought (from the docs) that __init__() would only be called when the module was loaded with using or `import?
The reason for the question is that I came across confusing errors when trying to precompile a package I am developing that has two extensions. I am trying to load the extensions whenever the needed dependencies are installed (without the user having to do using DependencyA, etc., explicitly) by having something like
function __init__()
try
Base.requires(Main, DependencyA)
catch
end
try
Base.requires(Main, DependencyB)
catch
end
end
In the main module of the package. It seems that the __init__() function of the main module of the package is being called during precompilation of both of the extensions. My current guess is that the error is something like a race condition in package precompilation - the error only happens when both extensions have been modified and so need to be precompiled at the same time.
I didn’t understand the error message, as __precompile__(false) is not declared anywhere in my code. [The Calling moment_kinetics.__init__() output is from a print statement in the __init__() function of the main moment_kinetics package that I added to see when it was being called].
Precompiling project...
✓ moment_kinetics
2 dependencies successfully precompiled in 16 seconds. 176 already precompiled.
1 dependency precompiled but a different version is currently loaded. Restart julia to access the new version
1 dependency had output during precompilation:
┌ moment_kinetics → manufactured_solns_ext
│ Calling moment_kinetics.__init__()
│ Calling moment_kinetics.__init__()
│ ┌ Warning: Module manufactured_solns_ext with build ID ffffffff-ffff-ffff-0000-1447d6c33442 is missing from the cache.
│ │ This may mean manufactured_solns_ext [246d111d-28e3-525f-a3fa-d7908d2bc032] does not support precompilation but is imported by a module that does.
│ └ @ Base loading.jl:1942
│ ┌ Error: Error during loading of extension manufactured_solns_ext of moment_kinetics, use `Base.retry_load_extensions()` to retry.
│ │ exception =
│ │ 1-element ExceptionStack:
│ │ Declaring __precompile__(false) is not allowed in files that are being precompiled.
│ │ Stacktrace:
│ │ [1] _require(pkg::Base.PkgId, env::Nothing)
│ │ @ Base ./loading.jl:1946
│ │ [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│ │ @ Base ./loading.jl:1806
│ │ [3] #invoke_in_world#3
│ │ @ Base ./essentials.jl:921 [inlined]
│ │ [4] invoke_in_world
│ │ @ Base ./essentials.jl:918 [inlined]
│ │ [5] _require_prelocked
│ │ @ Base ./loading.jl:1797 [inlined]
│ │ [6] _require_prelocked
│ │ @ Base ./loading.jl:1796 [inlined]
│ │ [7] run_extension_callbacks(extid::Base.ExtensionId)
│ │ @ Base ./loading.jl:1289
│ │ [8] run_extension_callbacks(pkgid::Base.PkgId)
│ │ @ Base ./loading.jl:1324
│ │ [9] run_package_callbacks(modkey::Base.PkgId)
│ │ @ Base ./loading.jl:1158
│ │ [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│ │ @ Base ./loading.jl:1813
│ │ [11] #invoke_in_world#3
│ │ @ Base ./essentials.jl:921 [inlined]
│ │ [12] invoke_in_world
│ │ @ Base ./essentials.jl:918 [inlined]
│ │ [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│ │ @ Base ./loading.jl:1797
│ │ [14] macro expansion
│ │ @ Base ./loading.jl:1784 [inlined]
│ │ [15] macro expansion
│ │ @ Base ./lock.jl:267 [inlined]
│ │ [16] __require(into::Module, mod::Symbol)
│ │ @ Base ./loading.jl:1747
│ │ [17] #invoke_in_world#3
│ │ @ Base ./essentials.jl:921 [inlined]
│ │ [18] invoke_in_world
│ │ @ Base ./essentials.jl:918 [inlined]
│ │ [19] require(into::Module, mod::Symbol)
│ │ @ Base ./loading.jl:1740
│ │ [20] include
│ │ @ Base ./Base.jl:495 [inlined]
│ │ [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│ │ @ Base ./loading.jl:2216
│ │ [22] top-level scope
│ │ @ stdin:3
│ │ [23] eval
│ │ @ Core ./boot.jl:385 [inlined]
│ │ [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│ │ @ Base ./loading.jl:2070
│ │ [25] include_string
│ │ @ Base ./loading.jl:2080 [inlined]
│ │ [26] exec_options(opts::Base.JLOptions)
│ │ @ Base ./client.jl:316
│ │ [27] _start()
│ │ @ Base ./client.jl:552
│ └ @ Base loading.jl:1295
└
I’m not certain, but if your extensions load the package which then loads them in __init__ via Base.require, it seems circular enough to cause problems at precompilation.
@Benny the extensions do import some function(s) from the package (which I guess is enough to load the package) - don’t extensions have to do that in order to do anything at all?
Strangely, adding the extension-triggering dependencies in the opposite order (i.e. something like Pkg.add(DependencyB); Pkg.add(DependencyA) instead of Pkg.add(DependencyA); Pkg.add(DependencyB)) seems to have made the error go away. Still very confused about what was causing it, so worried it might come back…
It’s their intended purpose, but I had to ask because you seemed surprised moment_kinetics was loaded by the extensions and because you’re already working outside the intended purpose. Extensions are intended to be conditionally loaded depending on what packages had been imported, not just installed, because source code imports are much easier to track than whether adding an arbitrary package also installs an extension dependency through some long dependency chain (why we compute Manifest.toml from Project.toml’s). Extensions aside, I’m also uncomfortable with the package loading other packages conditionally on their installation instead of making them dependencies, in which case the extension’s code would just be part of the package. I prefer conditions to be fully explicit or else varied very little like the operating system; project environments are neither.
Even if the error went away I wouldn’t trust that there isn’t a silent problem, especially with what seems like repeated loading (Base.require is documented to force reload), which could explain the “a different version is currently loaded” part. It’s still not clear what is happening, e.g. missing and mismatching names or nicknames (DependencyA), an absence of full protocols to replicate or evade the error, which involved modules import each other and in what order exactly, lacking precompilation reports for each add, misleading error message. I wonder if there is a way to log or print precompilation or load order more plainly.
Thanks @Benny! I think I’m starting to understand better. It makes sense I guess that init() is called when using MainPackage (or equivalent) is evaluated in the extensions when they are precompiled. I think I was a bit mislead by the docs for modules which say about __init__() that “Effectively, you can assume it will be run exactly once in the lifetime of the code” so I was confused by the multiple calls - I guess this is one of the cases that comes under ‘Effectively’ that people shouldn’t normally need to worry about.
Maybe it would help to expand a bit on the actual use case for doing what I’m doing. The main package is a simulation code, and I want to do something like: if NCDatasets is installed, then the user can set an option to request output in NetCDF files. But if the user doesn’t need NetCDF-format output, I don’t want to require having NCDatasets installed. Actually the bigger issue is my other optional dependency, Symbolics. I could live with always installing NCDatasets even if it’s not needed, but Symbolics is a very heavy dependency - it takes a long time to precompile, and adds a few minutes to the time it takes to build a system image (at least, it did on julia-1.9.x, this seems to be a lot better on 1.10.0!). Again though, the features that use Symbolics are called if the user sets certain options in an input file. The simulation code is usually run in batch jobs on HPC clusters rather than interactively, so to use extensions in the expected way, a bunch of different scripts would need to know whether NCDatasets and/or Symbolics are installed, load them if they are, and then load the simulation package. It seems simpler to handle that logic in one place.
So in the end what I want is something like ‘if NCDatasets is in Project.toml, load the extension MyNetCDFExt’. I can appreciate that allowing this in a way that’s compatible with the package that wants to do something like this just being a dependency of a dependency of something else, so none of the relevant packages need to be in Project.toml at all, could be challenging. I guess I’m thinking it would be nice to be able to do something like
pkg> Foo[OptionalFeature1,OptionalFeature2]
Or have in a Project.toml
[deps]
Foo[OptionalFeature1] = "xxxxxxxxxxxxxx"
And have the package load some extension-like things that provide extra features (but might have extra dependencies). That’s probably a complicated addition to the package management though!
Those needed updating, even before v1.9 introduced extensions, and the documentations on extensions aren’t entirely clear about the conditions for installation and precompilation. I still don’t really understand the exact process myself, so I’ve only done what I’ve seen to work.
This is not a feature and just doesn’t seem like it helps reproducibility. Imports are supposed to be how you control what is loaded, e.g. import moment_kinetics, NCDatasets to load moment_kinetics, NCDatasets, then MyNetCDFExt. The intended way for a package to “automatically” load another package is an import statement in its source code, making the 2nd package its dependency. I haven’t heard of any context where importing a package implicitly imports a separately installed package, even if it’s a specified weak dependency that may be automatically installed (not so in Julia). It seems possible but you need the conditional imports to occur after all installs, which I’m not sure can happen smoothly even in an ideal install order, something that is hard to reproduce and usually avoided by package managers. Maybe you can dodge these problems with a --compiled-modules=no process to disable precompilation, forcing package code to evaluate fresh in the process.