Ouch, it seems I’ve been bitten by the correctness issue in its following manifestation:
My package overloads the Base.split() for (::AbstractVector, ::Integer).
Unit tests pass, scripts/REPL work correctly.
However, when using the package in Pluto, I started to get incorrect results, silently, no errors or warning.
Debugging revealed that Pluto quietly loads the Lazy.jl package which also overloads Base.split(::AbstractVector, ::Any) which ended up being used instead of my method.
A type piracy has been committed. Unintended and undetected.
Isn’t this scary? A package developer has no control over the combination of packages his package will be used in. No amount of testing would help here.
I just downloaded Pluto and checked the manifest. There does not seem to be any dependency on Lazy.jl . This makes sense, because Lazy.jl is an unmaintained package (it’s last commit was 4 years ago).
So I think the solution is
Don’t overload methods on types you don’t own. That is, don’t commit type piracy.
Don’t use Lazy.jl, or other packages which commit type piracy. It’s not Pluto that’s causing a dependency on Lazy.jl, though. It’s something else.
Don’t overload methods on types you don’t own. That is, don’t commit type piracy.
Yep. Just checking that an existing function like Base.split(), has no specialisation for some type ::T (e.g., ::AbstractVector in my case), isn’t enough.
Don’t use Lazy.jl, or other packages which commit type piracy.
But how to ensure it? Get every package checked buy Aqua.jl?
It’s not Pluto that’s causing a dependency on Lazy.jl, though. It’s something else
Oh, my apology to Pluto.jl then for jumping to a conclusion here.
This is exactly the definition of type piracy. You don’t own the type AbstractVector (or Integer as mentioned in your first post), nor the function Base.split. So defining a method of that function with those types is type piracy.
You need to either define a new function (not a new method of a function in Base!), or a new type (ie a subtype of AbstractVector) that you can define a method for.
It shouldn’t be surprising that defining a new method for a function you don’t own, with types you don’t own, causes unexpected behaviour.
I agree. It may be not surprising in hindsight. But this “mistake” is very easy to make: both myself and the author of Lazy package, being carried away by Julia’s power of generic programming, committed the piracy with the intention to extend the Base function in a generic way, i.e., using an informal abstract interface like AbstractVector.
My concern is that this mistake occurs in silence. Shouldn’t be there a safe guarding mechanism that would warn about such ambiguities like methods clash?
There are some mitigating circumstances if the pirated definition is very natural and as such is essentially the only way to define something (think of Robin Hood, he wasn’t really a pirate ).
Base.split is only defined and used for strings. Another piece of advice is to never specialize a method if you’re making it serve some other purpose. For example, you should not overload Base.size(::TeeShirt) for your custom TeeShirt type to return shirt size, because Base.size is about the dimension of arrays.
Instead, only overload a method if you need code written by other people to operate on your custom type in a custom way (either because the existing version would not work or would be inefficient). For example, it’s common to extend Base.getindex to new matrix types because you need them to be usable like matrices in other code (e.g., in matrix multiplication code).
In this case, since nothing else exists that already provides this functionality (Iterators.partition is the closest I can think of), I would just rename your function to something else that is entirely new.
to me that sounds reasonable at a high-level, but I worry about the details. E.g. some packages are essentially allowed to pirate methods from another, since they are designed together to do so. Or packages like GenericLinearAlgebra that add linear algebra methods for bigfloats etc are piracy, but basically desired. So given there is some amount of desired piracy in the ecosystem, we wouldn’t want to necessarily require manual merges for every version of such packages. But then it gets more complicated to make an allowlist etc.
I wish there were a more explicit way to do this and report downstream piracy (privateering?) conflicts.
This is related to the traits issue highlighted in the viral instruction post (“what’s bad about Julia.”). Julia facilitates polymorphism through the workflow: (1) subtype some abstract type from another package (e.g. “AbstractArray”); (2) implement a list of methods for your new subtype (e.g. Base.size, Base.iterate, etc…). However, it is currently not at all clear what methods must be implemented if the downstream package’s struct is to work with other functions defined on the struct’s abstract supertype. There should be a way to declare method implementation requirements for subtypes of an abstract type (this would resemble the Holy Traits pattern) and it should be quarriable (e.g. with a function Base.methodrequirements(my_type)::Vector{Function}); packages that implement a subtype of a supertype with declared method requirements would throw a warning upon loading if the package does not define a required method. Similarly, there should be a way to declare functions that may be pirated while reporting duplicated piracy in downstream packages.
One of my favourite things about multiple dispatch was the freedom to choose those name for my function which other people are used to and are more likely to understand without help instructions. On this ground, I’d precisely overload Base.size(::TeeShirt) for my `TeeShirt’s, since the name is natural.
I was under impression that by allowing for that Julia community was creating a programming ecosystem governed rather by natural than a programming language.
).