Is it possible to precompile code in package extensions with PrecompileTools.jl?
Say I have a package Foo.jl with an extension FooMakieExt.jl for Makie.jl recipes. I would liike to write precompile workloads that would be triggered whenever a Makie.jl backend is loaded in the session.
I already use PrecompileTools.jl to precompile code in Foo.jl. But the question is how to deal with weak dependencies? Is there a way?
Extensions are precompiled separately and can import any of their parent packages’ dependencies including PrecompileTools, here’s an example: ModelingToolkit.jl/ext/MTKOrdinaryDiffEqBDFExt.jl at v11.26.8 · SciML/ModelingToolkit.jl · GitHub
Thank you @Benny , it worked for a package Core.jl that has a CoreMakieExt.jl extension. However, we have another package Main.jl with a MainMakieExt.jl extension that uses the recipe defined in CoreMakieExt.jl. How to inform PrecompileTools.jl that the recipe lives in the CoreMakieExt.jl?
In other words, can CoreMakieExt.jl be a dependency of MainMakieExt.jl for precompile workloads?
On most recent versions of Julia (there was a bug in early versions of 1.11 where this didn’t work), dependencies between extensions works “as expected”, or more formally if extension A’s triggers are a set of packages {A_i} and extension B’s triggers are {B_i}, and {A_i} is a strict subset of {B_i}, then A will be loaded before B.
For your usecase all you need to make sure is that Core is a strong dep of Main, or if not, then Core is declared as part of the extension triggers for MainMakieExt. Either way, that ensures that all the things needed for CoreMakieExt are triggered before the things needed for MainMakieExt.
(This is actually more general than precompile workloads: it means that extensions can generally rely on functionality defined in other extensions. The precompile workload isn’t anything ‘special’ in this regard – it’s just part of your extension.)
I want to address this wording in case there’s an underlying misconception, so pardon me if I misinterpreted: extensions are not intended to be dependencies i.e. we can’t import them or their internal names, nor can they import new dependencies that the trigger packages didn’t already have (otherwise, importing a set of packages could trigger an unbounded cascade of package imports that easily runs into impossible cycles). Packages are the dependencies, and penelopeysm explained how to arrange those to ensure an extension can rely on another.
Thank you all. You can find my attempt below, and it will clarify what I am trying to do:
The GeoTables.jl package depends on Meshes.jl (hard dep). The MeshesMakieExt has a viz recipe that is used by the viewer defined in GeoTablesMakieExt. I cannot precompile the viewer and the error message during precompilation indicates that viz is not defined in the precompilation workload:
Failed to precompile GeoTablesMakieExt [a9b04ea9-04f7-5d13-8c2e-9a13f647e8cf] to "/home/runner/.julia/compiled/v1.12/GeoTablesMakieExt/jl_jrwHUf".
ERROR: LoadError: MethodError: no method matching viz(::GridLayoutBase.GridPosition, ::Meshes.CartesianGrid{Meshes.𝔼{2}, CoordRefSystems.Cartesian2D{CoordRefSystems.NoDatum, Unitful.Quantity{Float64, 𝐋, Unitful.FreeUnits{(m,), 𝐋, nothing}}}, 2, Tuple{Unitful.Quantity{Float64, 𝐋, Unitful.FreeUnits{(m,), 𝐋, nothing}}, Unitful.Quantity{Float64, 𝐋, Unitful.FreeUnits{(m,), 𝐋, nothing}}}}; color::Observables.Observable{Any}, alpha::Float64, colormap::Symbol, colorrange::Symbol)
The function `viz` exists, but no method is defined for this combination of argument types.
Did you import a Makie.jl backend (e.g., GLMakie.jl, CairoMakie.jl) for visualization?
julia> using Meshes
julia> import GLMakie # or CairoMakie, WGLMakie, etc.
What am I missing?
My expectation is that loading GeoTables.jl with any Makie.jl backend will precompile the viewer. This is assuming that the viz recipe is already precompiled.
I’ve never made Makie recipes so I can’t really help with that. I wouldn’t expect a MethodError to be affected by precompilation, so my instinct is to separately try the routine in GeoTablesMakieExt’s workload in a REPL session without precompilation (which occurs in a child process outside our control), after loading everything it needs (trigger packages, other dependencies, extension files) and checking if :MeshesMakieExt exists and contains what you need with Base.get_extension. If the same MethodError with viz still happens during the routine, then you’d at least rule out precompilation being the obstacle. Maybe a MethodError would also print some closest matches to hint at incorrect dispatch intentions.
I’ve done that. Everything works as expected outside the precompilation workload.
What happens if you add the Base.get_extension(Meshes, :MeshesMakieExt) check and associated viz checks prior to the workload during GeoTablesMakieExt’s precompilation? println of the results should show up before the error is thrown.
I’ve added the following lines in the precompile.jl file as suggested:
using Meshes
Base.get_extension(Meshes, :MeshesMakieExt) |> println
The precompilation error shows nothing indicating that the MeshesMakieExt extension is not loaded. What else can I do to debug this further?
I understand that the extension should be available even though the only package triggering the extension in GeoTablesMakieExt is Makie.jl. Should Meshes.jl be added to the list of triggers somehow?
That confirms it then, we ran into a problem with the first half of:
where Core => Meshes, Main => GeoTables. I would indeed try putting Meshes in GeoTablesMakieExt’s trigger list. Note that this doesn’t imply that you make a GeoTablesMeshesMakieExt or demand users write using GeoTables, Meshes, GLMakie; Meshes is already loaded as a strong dependency of GeoTables.
How extensions are precompiled and loaded is not supposed to be exposed to the user, it should “just work” when we import the trigger packages, so it’s not documented for at least that reason. I’ll at least point out the merged PRs delay loading of extensions as much as possible · Pull Request #48674 · JuliaLang/julia and prevent loading other extensions when precompiling an extension · Pull Request #55589 · JuliaLang/julia, which means the current implementation loads all the packages first, but a precompiling extension will not start to load another extension even it ought to be there, exactly how GeoTablesMakieExt was missing MeshesMakieExt during precompilation but not during the REPL run.
The open issue Have extensions trigger from another extension getting loaded · Issue #48734 · JuliaLang/julia doesn’t exactly match your situation because it talks about extensions belonging to one package, but it is about ordering extensions loads. The linked merged PRs, which I think is still how it works, is to load extensions with fewer triggers earlier · Pull Request #49891 · JuliaLang/julia and Allow ext → ext dependency if triggers are a strict superset · Pull Request #56368 · JuliaLang/julia. Manually arranging trigger subsets should work, though the latter PR’s sentence “The same trick can also be used to make an extension in one package depend on an extension provided in another” seems to suggest that’s ideally unnecessary. If you want to notify the developers about your situation with GeoTables and a strong dependency Meshes each making an extension with Makie, I’d suggest commenting a sufficient summary on that open issue, especially if manually adding Meshes to GeoTablesMakieExt’s triggers does not work out.
Another reason for the lacking documentation is the design isn’t actually settled. Earlier I said
To clarify, that obstacle isn’t proven insurmountable, there are at least 2 open feature requests (can follow links from Issue #52663 · JuliaLang/julia) from the early release of extensions. Note that extensions loading more packages and triggering more extensions conflicts with PR #48674.
That is very helpful, thank you @Benny !
Unfortunately, I had to add a whole set of packages to make it work because the definitions of various functions live in extensions:
GeoTablesMakieExt = ["CategoricalArrays", "Colorfy", "Makie", "Meshes", "Unitful"]
I wonder if that solution is robust though, i.e., can survive future Julia releases. Should I merge the PR as is or wait for until the extension story is settled?
Was MeshesMakieExt not the only necessary strong dependency extension for GeoTablesMakieExt? Could you provide a full list (and also check get_extension for them during precompilation)? Not necessarily to me or here in particular, I’d definitely try to talk to Pkg extensions developers first because that list horrifies me, though the silver lining is you have something that works (successfully precompiles?). Knowing about all the extensions of dependencies helps, but that wouldn’t be necessary in a sane world; in a manner of speaking, strong and indirect dependencies ought to be implicit in the triggers list. I don’t know if that would be the most efficient way to implement the order of extensions loading.
Don’t. I’m just saying the implementation could still improve and features could still be added, which is true for almost all useful software. You have something that works, and AFAIK you’re not stuck with your long triggers list in future versions and can trim it to what the extension’s name implies after the extensions implementation improves.
The dependency graph for future reference:
Colorfy.jl → Meshes.jl → GeoTables.jl
Extensions:
Colorfy.jl → ColorfyUnitfulExt
Colorfy.jl → ColorfyCategoricalArraysExt
Meshes.jl → MeshesMakieExt
GeoTables.jl → GeoTablesMakieExt
In words:
Colorfy.jl tells how objects are transformed into colors. We implement some definitions as extensions for Unitful.jl and CategoricalArrays.jl.
Meshes.jl provides the viz recipe that places the colors at specific geometries in space. It is implemented in the Makie.jl extension.
GeoTables.jl adds a few interactive elements to the scene with the viewer and cbar. They are both implemented in the Makie.jl extension.
That does suggest another problem, assuming that all the extensions except GeoTablesMakieExt were able to precompile in separate tests. As far as I can see on the masters, all those extensions only have 2 triggers, so GeoTablesMakieExt getting 3 (itself, Makie, and manually adding Meshes) ought to have been enough to load the other extensions (of its strong dependencies) beforehand, per PR #49891.
I’ll note that Meshes has both Colorfy and Unitful as strong dependencies, but I don’t know if MeshesMakieExt depends on ColorfyUnitfulExt loading beforehand. That could be an issue down the road.
I am a bit skeptical to proceed with the merge of the PR at this point in time. The downstream visual regression tests still fail, this time accusing that Makie.jl is not loaded.
There is some complex interaction affecting the way these extensions are precompiled. I will watch the issue on GitHub and get back to this in the future.
Thank you @Benny for the insightful answers!
That’s also fair, you’re the one who has to maintain the extensions after all. I saw your comment on the Github issue, and while bearing in mind that you don’t have the time now to elaborate, “important comments” doesn’t really give the impression that you have an issue. Now that I think about it, your issue is standalone and reproducible, i.e. I can add the precompile commit, see the same error, and verify MeshesMakieExt was not loaded given a small edit, so it’s warranted to open a new issue in Pkg. There’s a chance it’d be considered a duplicate of a broader issue, but it’s also possible that it’s a smaller bug with a quicker fix.
When you do have the time, I encourage you to make a demo commit with a pre-workload check like:
println("Is MeshesMakieExt loaded? ", !isnothing(Base.get_extension(GeoTables, :MeshesMakieExt)) )
println("Number of viz methods: ", length(methods(viz)))
println()
and open an issue where you specify the commit, summarize that GeoTables depends on Makie but MeshesMakieExt (and its new methods) fails to be precompiled and loaded before GeoTablesMakieExt, and paste the error with builtin proof, only an add away.