Now that built in glue packages are on their way, I wonder if it is worthwhile to consider a new design pattern of optional multiple dispatch.
Typically in Julia we tend to take advantage of multiple dispatch rather than having multiple distinct functions with similar names in each package. For example, there is one Base.length(), and we tend to use multiple dispatch to overload this method for new types we introduce. Doing so however can lead to frustrating compilation issues such as method invalidation.
Alternatively, we could create FooPackage.length() and BarPackage.length(), but this would lead us to use namespace separation as is common in Python.
With glue packages, I see the potential for a middle ground between these two approaches.
The core package does no overloading of external methods and only provides types and methods confined to its module namespace. At the extreme, these types also do not inherit from any abstract types from outside the package.
Rather the task of overloading methods and extending types falls to an integration glue package. This glue package merely overloads methods from other packages or Base and then forwards the functions calls to their actual implementation within the core package namespace via inlining.
Some packages have already proceeded down this road and created “*Core” packages. In some cases, core packages consist mainly of type and their constructors and not much else. While this approach certainly is effective at preventing invalidations, it also means the “Core” package is practically useless on its own.
What I am asking for here is a clearer understanding of what a core package is. Can we design a core package that avoids the compilation issues associated with multiple dispatch but is still functional? Can we design a glue package that merely acts as a way to opt into the multiple dispatch paradigm?
At least in principle, this is an interesting idea, although I’d appreciate further clarity about your goals for this proposal: is it a TTL (time-to-load) optimization, or more of a design/namespace/package interaction proposal?
Currently (and hopefully this won’t be true forever), TTL is mostly due to invalidation checks, which come in two flavors: (1) what precompiled specializations in the new package need to be invalidated due to unanticipated methods defined in previously-loaded packages? (“backedge validation”), and (2) what specializations in previously-loaded packages need to be invalidated due to the new package? (“method insertion”) Specifically category 2 comes from your new package adding new methods to externally-defined functions. If you have relatively few of these, TTL should (for now) go down.
However, one thing I wonder about: how often are these new methods defined for functions owned by Base? Anything owned by Base presumably shouldn’t be made optional (it becomes type-piracy otherwise); if these account for the vast majority of method extensions, then I think your proposal would have little impact on TTL.
Again, we hope the dominance of TTL by these invalidation checks is a temporary situation that will be reduced in a future version of Julia (1.10 or higher). So I would urge some patience and suggest avoiding “unpleasant” distortions of the package ecosystem.
Multiple dispatch introduces complexity that does not exist in other languages. While it is nice to have, we do not have to use all the time. We may be able to simplify the design of some packages and and in turn simplify compilation.
From Chris’s write up and the advent of Core packages we see that reducing complexity is beneficial.
Concretely, it led to the creation of a package like StaticArraysCore.jl:
One issue I have this though is that StaticArraysCore.jl is actually not very useful by itself. I want to be able to calculate the length of a StaticArray without introducing the complexity of additional dependencies and potential invalidations.
My proposal is that we could move the definition of length to StaticArraysCore.jl but not import length from Base there. In StaticArrays.jl, we then define the multiple dispatch link.
By doing this, I can now use StaticArraysCore.jl and invoke StaticArraysCore.length() as needed without introducing the additional complexity introduced by using StaticArrays.jl.
The proposal is not inherently about time to load. It’s more about reducing some complexity of package design while making core packages more useful. The core packages will look more like a Python package and people could use as if it were one with namespace qualifications.
My proposal is that we could move the definition of length to StaticArraysCore.jl but not import length from Base there.
Then wouldn’t you need !(SVector <: AbstractVector)? Interfaces · The Julia Language says you need to be able to support certain methods, and that (indirectly, via size) includes Base.length.
Overall I’m pretty skeptical about this proposal. It seems to be about giving up on solving the expression problem. What’s the benefit? “Reducing complexity of package design” sounds nice, but what does that actually mean? Is it complex only if you’re planning to split between Core & non-Core?
Wouldn’t it be better to give up on the idea of *Core packages as an ugly bandaid and have someone spend three months improving the performance of package-loading? It seems likely to be less technical than other recent advances, and anyone comfortable with C could work on it. All it takes is 1-2 brave volunteers.
StaticArraysCore was (AFAIU) created so that people could extend methods to StaticArrays without having to depend on the full package and thereby did not have to take on the full load time cost. This is solved by package extensions in that you just make StaticArrays a weak dependency and add the extension methods into an extension module. So there is no need for Core packages anymore.
In the case of StaticArrays, how exactly would this work? My understanding is that extension packages are useful when you have two bits of functionality already loaded, and you want to make them interoperate seamlessly. So you’d still have a “main” package that defines things that are truly standalone, plus one or more extensions that get loaded conditionally. But:
Base.length, the example here, is always loaded because Base is always loaded. So this wouldn’t benefit from going in an extension package. Seems like anything extending a Base method should go in the main/core package.
A lot of the methods in StaticArrays extend functions owned by LinearAlgebra, but at present that’s always loaded too. Here the obvious solution is moving LinearAlgebra into a standalone package, but that appears nontrivial.
This is why I say the real problem is TTL. We don’t yet even have a clear sense of what kind of gains are possible, but until we try I think we’re doomed to an era of ugly hacks that work around the chains holding Julia back. Much better to just snip the chains and free Julia to be the language it should be.
The status quo if I import StaticArraysCore.jl right now is that SVector <: AbstractVector but SVector does not implement Base.length. This is exactly the issue I’m raising here. The core packages by themselves are in an inconsistent state right now. One cannot even display a SVector at with just StaticArraysCore.jl loaded.
julia> using StaticArraysCore
julia> sv = SVector{5, Int}((1,2,3,4,5));
julia> length(sv)
ERROR: MethodError: no method matching size(::SVector{5, Int64})
Closest candidates are:
size(::AbstractArray{T, N}, ::Any) where {T, N} at abstractarray.jl:42
size(::Union{LinearAlgebra.Adjoint{T, var"#s886"}, LinearAlgebra.Transpose{T, var"#s886"}} where {T, var"#s886"<:(AbstractVector)}) at C:\Users\kittisopikulm\.julia\juliaup\julia-1.8.5+0.x64.w64.mingw32\share\julia\stdlib\v1.8\LinearAlgebra\src\adjtrans.jl:173
size(::Union{LinearAlgebra.Adjoint{T, var"#s886"}, LinearAlgebra.Transpose{T, var"#s886"}} where {T, var"#s886"<:(AbstractMatrix)}) at C:\Users\kittisopikulm\.julia\juliaup\julia-1.8.5+0.x64.w64.mingw32\share\julia\stdlib\v1.8\LinearAlgebra\src\adjtrans.jl:174
...
Stacktrace:
[1] length(t::SVector{5, Int64})
@ Base .\abstractarray.jl:279
[2] top-level scope
@ REPL[16]:1
julia> applicable(size, sv)
false
Maybe the solution is to reintegrate the core packages eventually, but I’m not sure if we’ve solved the issues that led to their creation.
Maybe the solution is to reintegrate the core packages eventually, but I’m not sure if we’ve solved the issues that led to their creation.
My interpretation is that TTL is what led to their creation, so I am pretty sure that the answer is “no, we haven’t.” I’m just proposing that rather than make Julia-not-Julia, it would be better to address those issues instead.
In terms of the inconsistency, I’d say that at a minimum, what’s needed to support the AbstractArray interface needs to be in the core. I also think it’s piracy if any method extending a function owned by Baseisn’t in the core, but piracy is sanctioned in cases where the justification is sufficiently strong.
Beyond that, I think it’s better if the package ecosystem avoids expensive gyrations to circumvent a short-term problem, and just focuses on fixing that problem ASAP instead.
For example, OrdinaryDiffEq.jl changed dependency to StaticArraysCore.jl I believe. This improved loading time of OrdinaryDiffEq.jl, but basically it just shifts the “blame” from OrdinaryDiffEq.jl devs to StaticArrays.jl devs. Did this have a big effect on the end user who has to load StaticArrays.jl anyway in order to create and pass SVectors to OrdinaryDiffEq.jl?
The problem right now though, as was noticed above, is that end users can depend on StaticArraysCore, and thus create SVectors without loading StaticArrays.jl. It would have worked better if only other packages could depend on StaticArraysCore. Right now basically all of StaticArrays type-piracy.