Use both Requires (pre-1.9) and Extensions, using isdefined(Base, :get_extension) to determine which system to use
Put the conditional dependencies in [weakdeps] and [extras] and [test] sections
Extensions go in ext/ExtensionExt.jl, with a structure like:
module UnitfulExt
import DiffEqBase
isdefined(Base, :get_extension) ? (import Unitful) : (import ..Unitful)
# ... extensions here
end
(note that Requires plops the dependencies into the parent module, so you use import ..Dependency, whereas the builtin extensions support just makes it happen)
Requires support for older versions has this __init__ for the parent package:
function __init__()
@static if !isdefined(Base, :get_extension)
@require Unitful="1986cc42-f94f-5a68-af5c-568840ba703d" begin
include("../ext/UnitfulExt.jl")
end
end
end
Might be good to distill this into a “worked example” of using extensions while supporting older Julia versions. Also wonder if any of this could be streamlined a bit now that it works
This is more a feature request than a docs one, but one case which hasn’t been covered is how to migrate an existing glue package into an extension. Some of these packages have their own dependencies separate from the weak deps, so those are probably out. Others are large enough that converting them into extensions with a Requires.jl fallback for <1.9 isn’t feasible because one misses out on precompilation. The only complete suggestions I’ve seen so far for this are either to fork one’s codebase into pre-1.9 and post-1.9 versions, duplicate code between extension and glue package, or to wait for a Julia version with package extensions to become LTS.
I think we may have covered this in a Slack discussion but haven’t put it online yet? The answer is that it’s a missing feature right now. The posted way in Slack to do this was to make the glue package be a dependency of the original package and then make the extension module reexport using the glue package. That sounded great, but when I went to try and go do that, I realized it did not work because it introduced a circular dependency, and such circular dependencies are not allowed in Pkg.
To bring this down to something more concrete, DiffEqBase is the core of differential equations, Zygote does automatic differentiation, and SciMLSensitivity adds adjoint overloads to the differential equation solvers for automatic differentiation. Right now if you try to use Zygote on a differential equation without using SciMLSensitivity, you get an error telling you to using SciMLSensitivity, so the improvement would be to use the package extension functionality to automatically load SciMLSensitivity when Zygote and DiffEqBase are both loaded. The suggested way to do this was to make SciMLSensitivity into a dependency of DiffEqBase and then make a ZygoteExt whose only code is @reexport using SciMLSensitivity. This would make SciMLSensitivity get installed even when it isn’t needed, but it would only make it get using’d as required so it’s a partial solution.
But what I’m saying the issue is, is that if you try and make SciMLSensitivity into a dependency of DiffEqBase you get an error about having a circular dependency because (as a glue package) it has a dependency on DiffEqBase. @ToucheSir I know there’s the similar case of NNLibCUDA: there’s NNLib.jl and CUDA.jl and when both are in the environment you’d want to pull in NNLibCUDA.jl. But NNLibCUDA.jl will have a dependency on NNLib, so that will fail. If the circular dependency doesn’t exist then that would be a solution though.
But that’s at least my current understanding of what @kristoffer.carlsson was suggesting, maybe I wasn’t understanding it correctly or he has a nice way to get around that limitation.
Note that it doesn’t have to be named like like that. Although the extension module is in most cases invisible to users, it might show up in e.g. stacktraces. I think a more descriptive name might be useful, for example DiffEqBaseUnitful for DiffEqBases extension to Unitful. (I can imagine many package having StaticArraysExt for example, and perhaps it might be confusing in some cases.)
Note that an extension can be conditional on more than just one package.
@ChrisRackauckas We covered parts but not all of this across a few different Slack threads (I think you weren’t on all of them), so I figured it would be best to put it in one public place to get more thoughts.
The extension package has deps issue is actually the least worrisome of the two for me. e.g. for the NNlibCUDA example, that’s already a hard dep of Flux so we could easily follow the hard dep → weak dep migration path outlined in the docs (@fredrikekre we want to explicitly avoid being conditional on the glue package here because it’s, well, a glue package).
The bigger challenge I see is the second one, which is what happens when you have a library with multiple “backends” represented as extensions that weren’t previously hard dependencies. I tried this in Re-integrate NNlibCUDA as a package extension by ToucheSir · Pull Request #445 · FluxML/NNlib.jl · GitHub, but it turns out the precompilation hit from Requires is so large that it negates any benefit of using extensions pre 1.9. After a long back and forth on Slack, it didn’t seem like anyone had a reliable “polyfill” for this that enabled precompilation on <1.9, and so I stopped looking into it altogether.
Good point, and as a point of style we should probably stick to a standard rule. I think XY where X is the package that is being extended and Y is the trigger/glue is a pretty good rule, like what you show with DiffEqBaseUnitful. We should probably rename our extensions to match that.
Personally I would say “meh” about working too hard for Julia versions prior to 1.9. Packages should function correctly on all Julia versions, but if a recent Julia version provides the best experience I don’t think we need to kill ourselves trying to match it on older versions, since we can just tell people to use the latest release. (Yes, I know 1.9 is not yet released…)
If you want to make changes that optimize the 1.9+ experience but significantly degrade the experience on older Julia versions, to me that sounds like time for a new minor (or breaking, if necessary) release and minimum bounds on the Julia version. In weird cases you might need RetroCap or something. That will allow you to keep users on the older versions until they adopt a Julia version optimized for the new design.
In my view we should play the long game, rather than going to a lot of effort to make improvements that will have a transient lifetime.
Thanks Tim. Given we still receive issues from users on 1.7.x (including from orgs using Julia. I presume upgrading production systems and/or clients takes time) and Flux’s previous less-than-stellar experience using backport branches, it looks like this is going to be a longer and more incremental transition than we originally expected. If anyone is interested, I can try to keep this topic updated as we make progress.
I find the conditional on imports cumbersome (duplicate list). How can we avoid these kind of duplicates:
@static if isdefined(Base, :get_extension)
import Unitful: Quantity, RealOrRealQuantity, ustrip, unit
else
import ..Unitful: Quantity, RealOrRealQuantity, ustrip, unit
end
If I have [weakdeps] and [extensions] sections in my Project.toml, what happens if I load my package under an older Julia? The implicit answer from what I read is that just would be ignored and cause no errors, but could somebody tell me explicitly?
My possible use case: The main functionality of the package could stay LTS compatible and needs just minimal dependencies, but there would be some special subpackage, using a lot of dependencies, to be run under Julia >= 1.9
The weak dependencies will be ignored, but you still need to list them in the [extras] section, otherwise the compat specification won’t be intelligible to older versions of Julia. Older versions of Julia only look at the [deps] section for dependencies
It is also used for packages that otherwise are indirectly dependencies (like packages you need in tests), here ones that you would add with Requires.jl, which is the fallback of pre-extension-time one should fall back to if on Julia 1.8 or less.
