But I wondered how this works for people with a lower Julia version.
What happens if a Project.toml has these [weakdeps] and [extensions] sections, but your Julia version is 1.6, or 1.0?
(1.0 is still supported by Distributions.jl)
Seems to be a line about that in the dev docs you linked.
If you want to use an extension with compatibility constraints while supporting earlier Julia versions you have to duplicate the packages under [weakdeps] into [extras]. This is an unfortunate duplication but without doing this the project verifier under older Julia versions will complain (error).
I had a bit of trouble figuring it out the first time, but now I have an example of it completed:
It’s very formulaic to follow that and just move unitful support to extensions. What DiffEqBase is doing supports both Requires and extensions, and which one it uses is dependent on whether it’s Julia v1.9 or not.
So backwards compatible, formulaic, and only a few minutes? People should start slapping extension packages everywhere!
Use both Requires (pre-1.9) and Extensions, using isdefined(Base, :get_extension) to determine which system to use
Put the conditional dependencies in [weakdeps] and [extras] and [test] sections
Extensions go in ext/ExtensionExt.jl, with a structure like:
module UnitfulExt
import DiffEqBase
isdefined(Base, :get_extension) ? (import Unitful) : (import ..Unitful)
# ... extensions here
end
(note that Requires plops the dependencies into the parent module, so you use import ..Dependency, whereas the builtin extensions support just makes it happen)
Requires support for older versions has this __init__ for the parent package:
function __init__()
@static if !isdefined(Base, :get_extension)
@require Unitful="1986cc42-f94f-5a68-af5c-568840ba703d" begin
include("../ext/UnitfulExt.jl")
end
end
end
Might be good to distill this into a “worked example” of using extensions while supporting older Julia versions. Also wonder if any of this could be streamlined a bit now that it works
This is more a feature request than a docs one, but one case which hasn’t been covered is how to migrate an existing glue package into an extension. Some of these packages have their own dependencies separate from the weak deps, so those are probably out. Others are large enough that converting them into extensions with a Requires.jl fallback for <1.9 isn’t feasible because one misses out on precompilation. The only complete suggestions I’ve seen so far for this are either to fork one’s codebase into pre-1.9 and post-1.9 versions, duplicate code between extension and glue package, or to wait for a Julia version with package extensions to become LTS.
I think we may have covered this in a Slack discussion but haven’t put it online yet? The answer is that it’s a missing feature right now. The posted way in Slack to do this was to make the glue package be a dependency of the original package and then make the extension module reexport using the glue package. That sounded great, but when I went to try and go do that, I realized it did not work because it introduced a circular dependency, and such circular dependencies are not allowed in Pkg.
To bring this down to something more concrete, DiffEqBase is the core of differential equations, Zygote does automatic differentiation, and SciMLSensitivity adds adjoint overloads to the differential equation solvers for automatic differentiation. Right now if you try to use Zygote on a differential equation without using SciMLSensitivity, you get an error telling you to using SciMLSensitivity, so the improvement would be to use the package extension functionality to automatically load SciMLSensitivity when Zygote and DiffEqBase are both loaded. The suggested way to do this was to make SciMLSensitivity into a dependency of DiffEqBase and then make a ZygoteExt whose only code is @reexport using SciMLSensitivity. This would make SciMLSensitivity get installed even when it isn’t needed, but it would only make it get using’d as required so it’s a partial solution.
But what I’m saying the issue is, is that if you try and make SciMLSensitivity into a dependency of DiffEqBase you get an error about having a circular dependency because (as a glue package) it has a dependency on DiffEqBase. @ToucheSir I know there’s the similar case of NNLibCUDA: there’s NNLib.jl and CUDA.jl and when both are in the environment you’d want to pull in NNLibCUDA.jl. But NNLibCUDA.jl will have a dependency on NNLib, so that will fail. If the circular dependency doesn’t exist then that would be a solution though.
But that’s at least my current understanding of what @kristoffer.carlsson was suggesting, maybe I wasn’t understanding it correctly or he has a nice way to get around that limitation.
Note that it doesn’t have to be named like like that. Although the extension module is in most cases invisible to users, it might show up in e.g. stacktraces. I think a more descriptive name might be useful, for example DiffEqBaseUnitful for DiffEqBases extension to Unitful. (I can imagine many package having StaticArraysExt for example, and perhaps it might be confusing in some cases.)
Note that an extension can be conditional on more than just one package.
@ChrisRackauckas We covered parts but not all of this across a few different Slack threads (I think you weren’t on all of them), so I figured it would be best to put it in one public place to get more thoughts.
The extension package has deps issue is actually the least worrisome of the two for me. e.g. for the NNlibCUDA example, that’s already a hard dep of Flux so we could easily follow the hard dep → weak dep migration path outlined in the docs (@fredrikekre we want to explicitly avoid being conditional on the glue package here because it’s, well, a glue package).
The bigger challenge I see is the second one, which is what happens when you have a library with multiple “backends” represented as extensions that weren’t previously hard dependencies. I tried this in Re-integrate NNlibCUDA as a package extension by ToucheSir · Pull Request #445 · FluxML/NNlib.jl · GitHub, but it turns out the precompilation hit from Requires is so large that it negates any benefit of using extensions pre 1.9. After a long back and forth on Slack, it didn’t seem like anyone had a reliable “polyfill” for this that enabled precompilation on <1.9, and so I stopped looking into it altogether.
Good point, and as a point of style we should probably stick to a standard rule. I think XY where X is the package that is being extended and Y is the trigger/glue is a pretty good rule, like what you show with DiffEqBaseUnitful. We should probably rename our extensions to match that.
Personally I would say “meh” about working too hard for Julia versions prior to 1.9. Packages should function correctly on all Julia versions, but if a recent Julia version provides the best experience I don’t think we need to kill ourselves trying to match it on older versions, since we can just tell people to use the latest release. (Yes, I know 1.9 is not yet released…)
If you want to make changes that optimize the 1.9+ experience but significantly degrade the experience on older Julia versions, to me that sounds like time for a new minor (or breaking, if necessary) release and minimum bounds on the Julia version. In weird cases you might need RetroCap or something. That will allow you to keep users on the older versions until they adopt a Julia version optimized for the new design.
In my view we should play the long game, rather than going to a lot of effort to make improvements that will have a transient lifetime.
Thanks Tim. Given we still receive issues from users on 1.7.x (including from orgs using Julia. I presume upgrading production systems and/or clients takes time) and Flux’s previous less-than-stellar experience using backport branches, it looks like this is going to be a longer and more incremental transition than we originally expected. If anyone is interested, I can try to keep this topic updated as we make progress.
I find the conditional on imports cumbersome (duplicate list). How can we avoid these kind of duplicates:
@static if isdefined(Base, :get_extension)
import Unitful: Quantity, RealOrRealQuantity, ustrip, unit
else
import ..Unitful: Quantity, RealOrRealQuantity, ustrip, unit
end
