"Meaning", type-piracy, and method merging

[jlperla]

The meaning of “meaning”
A common theme that comes up when discussing type-piracy/method merging, etc. is that there “can only be one meaning” for a given function without namespace qualifications (I am being careful there to say function, not method). I just want to get some clarification from the developers on this, because clarity in communication would help for educating users, deciding what macros are reasonable, and what can be discouraged. (Of course, part of this is driven by the fact that most operators need to be in Base to be practical, and that a large and expanding Base and standard library ends up laying claim to a large number of functions.)

First, I would love to better understand what “meaning” means. What I believe you have in mind is that there is an expectation that the given “function” has a set of methods that operate on a set of types, which themselves are expected to have other methods. This is informally defined and unenforced, though it may be documented (e.g. https://docs.julialang.org/en/stable/manual/interfaces/) If so, then I would say that this sense of “meaning” is roughly equivalent to concepts (unenforced pre C++20) in C++, typeclasses in Haskell, etc.

Now, in this way of thinking (which is that there is some collection of abstract types and methods, informally enforced) “meaning” is related to both the function and the hiearchy of types which dispatch to methods within the function. For example, looking the size function in base:

help?> size
search: size sizeof sizehint! Csize_t resize! filesize Cssize_t displaysize @nospecialize

  size(A::AbstractArray, [dim...])

  Return a tuple containing the dimensions of A. Optionally you can specify the dimension(s) you want the length of,
  and get the length of that dimension, or a tuple of the lengths of dimensions you asked for.

This tells me that in Base the only “meaning” of the function size is something defined on AbstractArray <: Any that has a particular interface associated with it. Of course, if I create MyType <: AbstractArray then it would be a terrible idea for me to not have it follow this interface (unlike Concepts in C++ this is not enforced… but that is totally fine).

What exactly is “Type Piracy”?

Now, what would happen if I wanted to define a size method on a completely unrelated set of types? In particular

abstract type NothingLikeAbstractArray end
type ConcreteNotAnAbstractArray <: NothingLikeAbstractArray  end
Base.size(x::ConcreteNotAnAbstractArray) = 1

But note that NothingLikeAbstractArray <: AbstractArray == false. There is no overlap of types whatsoever with the existing “meaning” of size. It is unambiguous and there is no way whatsoever that the different “meaning” (i.e., the AbstractArray methods) would ever be involved in dispatch. In can happily live in parallel.

This sort of thing has been called “type-piracy” or perhaps “mild type-piracy” and discouraged in the interest of having only a single “meaning” for size, but I don’t see what is wrong with it?

Another example, in DifferentialEquation.jl there is a solve function with a particular “meaning”. The typical signature is along the lines of solve(prob::DiffEqBase.AbstractODEProblem{uType,tType,isinplace},...)

But what is wrong with me defining:

type MyModelNotRelatedToODE <: Any end
solve(mod::MyModelNotRelatedToODE, ...)

Again, this can happily live in parallel with the meaning of solve in DifferentialEquations.jl, even without namespace qualifications, since MyModelNotRelatedToODE <: AbstractODEProblem==false, etc. If namespaces are keeping these “meanings” separate, then they are getting in the way of simple code in this case.

My Definition of Type Piracy:
Take as given a method f(x::A) then if someone else were to define:

• f(x::B) for B <: A = Pirate!
• f(x::C) for (C >: A) && !(C <: A) = Not a Pirate!

So, given size(x::AbstractArray) and AbstractArray <: Any then defining

• size(x::DenseVector) = Pirate!
• size(x::MyType) for MyType <: Any as long as MyType >: AbstractArray == false = Not a Pirate!
• size(x::Any) = Not a Pirate! (this is because it is not possible to influence the dispatch for the AbstractArray types)

Should Multiple Meanings Be Discouraged?:

Why is this important? Because there are a lot of function names which are useful in different contexts, and it would be perfectly find to have using for them all. Furthermore, I would guess that the vast majority of cases being discouraged as “type piracy” are “not a pirate!” according to my definition, where there is no issues at all having parallel meanings. In my mind, the idea “type-piracy” is very real, but it is only meaningful if you write methods which change where existing types get dispatched.

In particular, people coming from single-dispatch languages would be almost never be doing that - in part because they don’t even know about generic programming. In virtually all those cases, they would write type MyType <: Any and then dispatch f(x::MyType). Unless f(x::Any) was already created in a package - which would be a lousy package design - this is not type-piracy according to my definition.

Why Not Manually Merge by Declaring Methods in Existing Namespaces?
Lets say that we no longer discourage the “Not a Pirate!” scenarios above, then why not just have everyone manually merging them? The short answer is the fragility or the ordering of method definitions.

That isn’t to say that automatic method merging is necessarily the best approach, but it suggests there is no reason that multiple reasonable different “meanings” on disjoint types in the same namespace. Having a clean way to manually merge, and having it clarified in the core documents, would make this much cleaner. Now you may say “just use import instead of using” but that isn’t possible with operators, it isn’t possible with functions already defined in Base, and (in the examples above) it is inconvenient without introducing any sort of “safety”.

[mbauman]

https://docs.julialang.org/en/stable/manual/style-guide/#Avoid-type-piracy-1

“Type piracy” refers to the practice of extending or redefining methods in Base or other packages on types that you have not defined.

You’re reaching out of your module and messing with someone else’s. Meaning doesn’t really have any bearing on whether something is doing piracy or not.

What you’re considering here isn’t really piracy at all — it’s what we’ve typically called “puns.” That is, you’re using the same function to do something completely different, hence pun: same word, different meaning. It wasn’t initially obvious that puns would be an impediment to generic programming. There were initially a number of cases built into base that re-used the same function for multiple purposes. The biggest problem with puns is that folks need to be explicitly aware of their function signatures. It breaks duck-typing.

While the documentation for size doesn’t show it, we have a number of array-like objects that implement size as you might expect. For example numbers are able to behave like a 0-dimensional array. I’d probably be good to remove the ::AbstractArray from that definition.

[Nosferican]

The danger of type-piracy is that code can behave in ways that can be unexpected or unclear.
For type-piracy to occur a necessary condition is to have multiple namespaces (e.g., Package A and Package B). As usual, there are two ways piracy can occur in a dispatch language
(i.e., function or struct).

1. First case (StatsBase)

"""
    coef(obj::StatisticalModel)
Return the coefficients of the model.
"""
coef(obj::StatisticalModel) = error("coef is not defined for $(typeof(obj)).")

Code will look like:

using StatsBase: StatisticalModel
import StatsBase: coef
mutable struct MyModel <: StatisticalModel
    β::Vector{Float64}
end
coef(obj::MyModel) = getfield(obj, :β)

in this case, this is not type piracy since MyModel and coef(::MyModel) is defined in my package. It doesn’t matter that MyModel <: StatisticalModel

2. Extension of function to another object (pirate way)

import Base: step
obj = 1:3 # StepRange{Int64,Int64}
step(obj) # 1
obj2 = collect(obj)
"""
    step(obj::AbstractVector{<:Number})
Return the minimum distance between elements.
"""
step(obj::AbstractVector{<:Number}) = minimum(diff(obj))

Here step is generalizing that applies to AbstractRange to AbstractVector{<:Number} which are both defined in foreign namespaces. This is discouraged even if the function gives the same idea or whatnot.

3. Non-piracy versions of (2)

Using a wrapper,

import Base: step
struct LikeRange
    data::Vector{Int64}
end
obj = LikeRange(collect(1:3))
"""
    step(obj::LikeRange)
Return the minimum distance between elements.
"""
step(obj::LikeRange) = minimum(diff(obj.data))

or defining a new function

obj = collect(1:3)
step2(obj::AbstractVector{<:Number}) = minimum(diff(obj))

[rdeits]

@jlperla

This sort of thing has been called “type-piracy” or perhaps “mild type-piracy” and discouraged in the interest of having only a single “meaning” for size , but I don’t see what is wrong with it?

This is not correct. Well, I’m sure people have called what you’re describing “type piracy”, but that’s not the generally agreed-upon definition. Defining a method of someone else’s (i.e. Base’s) function on your type is fine. That’s not type piracy. Type piracy is defining a method of someone else’s function on someone else’s type.

Take as given a method f(x::A) then if someone else were to define:
f(x::B) for B <: A = Pirate!

Again, this isn’t type piracy, as specified in the manual: https://docs.julialang.org/en/stable/manual/style-guide/#Avoid-type-piracy-1

You’re, of course, welcome to use whatever you want to mean whatever you want, but in the interest of clarity, it will be easier to have a discussion if we’re all speaking the same language.

[jlperla]

@mbauman

You’re reaching out of your module and messing with someone else’s. Meaning doesn’t really have any bearing on whether something is doing piracy or not.

Thank you, that is helpful and precise definition of type piracy. Of course, there are cases (particularly any operators defined in Base or other packages you want to use) where you have no choice but to pirate.

Let me rename “pirate” vs. “not a pirate” to “bad-pun” vs. “just-a-pun”.

Do I have what you have in mind for “meaning” correct?

@mbauman

The biggest problem with puns is that folks need to be explicitly aware of their function signatures. It breaks duck-typing.

For sure, I can see where a “bad pun” according to my definition would break plenty of stuff. But you give an example of where one of my “just-a-pun” breaks duck-typing in a reasonable scenario used in Base? If you are saying that size has a method defined on Number or even Any then that just changes where the “bad-pun” line starts. Julia doesn’t really do the full-on generic duck-typing that I think of (e.g. C++ has tricks to dispatch based on predicates about properties/methods of the type rather than relying on its supertype).

I think this would be very useful for documenting when and where punning is a problem.

@mbauman

You’re reaching out of your module and messing with someone else’s. Meaning doesn’t really have any bearing on whether something is doing piracy or not.

The issue is that these interact, unless I am missing something, because in order to conveniently “pun” in the current namespace setup, you need to pirate. Going back to my examples (and leaving aside the duck-typing issues) how many cases of type-piracy are just people trying to make “just-a-pun” examples convenient? I am willing to bet that the use of type-piracy for either “bad-puns” is pretty rate.

If there were utility macros to make merging methods for include and function definitions more convenient, then they would have the effect of people declaring their puns explicitly instead of manually doing type-piracy? Whether you like puns or not, it makes things more .grep-abla

[jlperla]

@rdeits

You’re, of course, welcome to use whatever you want to mean whatever you want, but in the interest of clarity, it will be easier to have a discussion if we’re all speaking the same language.

No, I want to use the same language as everyone, I have just been confused by a lot of the discussion on it. What exactly is being discouraged, etc. From what it sounds like, it is “punning” that is the question I am getting at. Of course, piracy is often the consequence of punning, which is why I want to make sure I understand what the issues are…

[jlperla]

@Nosferican

The danger of type-piracy is that code can behave in ways that can be unexpected or unclear.

For sure, there are many classic examples. I understand those in the examples you are giving (that are all in my “pirate!” or “bad-pun” examples). It appears that my question is one of “punning” with merging into the same namespace.

[rdeits]

I think I would agree that your issue is more with punning. And, just to be clear, you said above that:

Thank you, that is helpful and precise definition of type piracy. Of course, there are cases (particularly any operators defined in Base or other packages you want to use) where you have no choice but to pirate.

which is still not true. Defining a Base method on your own type is perfectly fine and not type piracy at all. It cannot possibly affect the behavior of anyone’s code unless they are using your type, so there’s no piracy issue.

For example, this code does not exhibit type piracy:

struct Bar
end

Base.:+(b1::Bar, b2::Bar) = 10

because I’ve only changed the behavior of + for my own type. Anyone not using my new Bar type is guaranteed to be unaffected. And anyone who is using my Bar type has, by using my code, essentially opted in to the behaviors I’ve defined for my particular types.

On the other hand, you can certainly argue that my definition of + is nonsensical or a bad pun. And I would agree with you! But the issue is one of clear meaning and expectations, not one of type piracy.

[jlperla]

@mbauman

You’re reaching out of your module and messing with someone else’s. Meaning doesn’t really have any bearing on whether something is doing piracy or not.

@rdeits

which is still not true. Defining a Base method on your own type is perfectly fine and not type piracy at all.

(Just wanted to point out this seems slightly out of sync with @mbauman definition above)

OK, so based on this definition, if I define + on MyNumber <: Number it is not type piracy, and if was to add in a package to extend Distributions.jl in the Distributions namespace with a function called “expectations” operating on the various types in Distributions.jl, then it would not be type piracy.

If so, then it sounds like “Type Piracy” should almost always be discouraged or treated with a great deal of caution.

So lets say that the “biggest” supertype for any method on size is Number <: Any and I define a function Base.size(x::MyType) where MyType is not a subtype of Number. then this is not type piracy (which is a good check on one of the things to discourage). In this case, while the Base may not be for type piracy, it is being done to make punning convenient. In the case of size we know it will be in Base and that using Base is always there (in practice), so it is easy to know where to put the method. It is not fragile to any ordering. But that is not the case of doing punning with other packages, e.g. the solve in DiffEq, where the ordering matters.

But I would still prefer to say
@merge function solve(x::MyType) or @merge size(x::MyType) = .... to merge it into the active namespace (if it exists) or the @merge using MyPackage if I had my function in a package. I think it is honest on what you are doing, and makes punning a lot cleaner. Again… all of this only applies to to cases where things are not a “bad-pun”, which is why I am not suggesting things are automatic.

[mbauman]

@jlperla

(Just wanted to point out this seems slightly out of sync with @mbauman definition above)

Nope, we’re in agreement. There’s an implicit AND within my statement that I probably could have made clearer. Type piracy requires two things:

• You’re extending a function you don’t own AND it’s
• On a signature that will catch a set of types that are totally disjoint from the types you define.

[rdeits]

@jlperla

If so, then it sounds like “Type Piracy” should almost always be discouraged or treated with a great deal of caution.

Definitely agreed.

OK, so based on this definition, if I define + on MyNumber <: Number it is not type piracy, and if was to add in a package to extend Distributions.jl in the Distributions namespace with a function called “expectations” operating on the various types in Distributions.jl, then it would not be type piracy.

Yup and yup (assuming that expectations is your package’s function).

So lets say that the “biggest” supertype for any method on size is Number <: Any and I define a function Base.size(x::MyType) where MyType is not a subtype of Number. then this is not type piracy (which is a good check on one of the things to discourage).

The subtyping doesn’t really enter into it. It’s not type piracy even if MyType <: Number, by the same argument you made above, because you own MyType.

In this case, while the Base may not be for type piracy, it is being done to make punning convenient.

I would argue that Base doesn’t commit type piracy, because it’s just defining its own methods on its own types, which is perfectly acceptable for any other module to do. But Base is a bit special anyway, so this probably isn’t an important point.

[rdeits]

Just to show that we’re not kidding around about type piracy being a bad idea, try this:

julia> Base.:+(::Int, ::Int) = 2

fatal: error thrown and no exception handler available.
UndefRefError()
rec_backtrace at /buildworker/worker/package_linux64/build/src/stackwalk.c:84
record_backtrace at /buildworker/worker/package_linux64/build/src/task.c:245
jl_throw at /buildworker/worker/package_linux64/build/src/task.c:564
jl_f__apply at /buildworker/worker/package_linux64/build/src/builtins.c:413
stacktrace at ./stacktraces.jl:151
display_error at ./client.jl:126
unknown function (ip: 0x7fee69293c0d)
jl_call_fptr_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:339 [inlined]
jl_call_method_internal at /buildworker/worker/package_linux64/build/src/julia_internal.h:358 [inlined]

Julia crashes immediately because, as it turns out, addition of integers is pretty important and is used in lots of places internally in the language.

On the other hand,

julia> struct Foo
       end

julia> Base.:+(::Foo, ::Int) = 2

is perfectly fine (i.e. not type piracy). Base never uses your Foo type, so nothing it does is affected until you actually hand a Foo object to a Base function.

[jlperla]

@rdeits

Just to show that we’re not kidding around about type piracy being a bad idea,

I need no convincing! Your precise definition of type piracy was enough for me, and I find it hard to think of examples where it would be a good idea. That said, it also seems like it is unlikely to be accidentally stumbled upon by new users (and more likely to be done by users who either really know what they are doing, or think they do).

People seem to throw around the word “type piracy” in many circumstances that don’t meet your criteria, so I am glad I asked for a clarification. I have also heard the term “mild type piracy” when critiquing what I would now call “punning”.

But back to that: do I have your sense of “meaning” right? And (in cases where it is not type-piracy) what is wrong with punning on disjoint sets of types? As @jandehaan shows, you need to be very careful about automatic conversions from base-types, but the main use of punning - especially from those coming from single-dispatch worlds - is on sets of types with no overlap in the dispatching.

You mentioned that it breaks convenient ducktyping? I don’t understand why the following should be discouraged. Take f(x::A) then if someone else to merge in:

• f(x::B) for B <: A = Bad Pun!
• f(x::C) for C >: A= Bad Pun! (because they might havef(x::B)` dispatch on their type without their knowledge. Is this the duck-typing one you are worried about?
• f(x::D) for !(D <: A || D>: A) = Maybe not so bad.

Of course, if I merge a method into a function where A == Any, then every pun is bad, but there are plenty of cases

[jlperla]

@ChrisRackauckas

That can only happen if someone knows the existence of both packages.

Agreed, it is a really neat example, but I also think it may be more instructive as an example of why broad conversions and promotions with abstract base types are dangerous (as I am pretty sure it doesn’t follow the type-piracy definition described above.)

[ScottPJones]

@jlperla

And (in cases where it is not type-piracy) what is wrong with punning on disjoint sets of types?

It can be an issue with readability.

The problem that I’ve seen in practice, is that if you are looking at a piece of generic code, and you see:
a * b ^ c, you have zero idea what that is actually doing. It could be concatenating string a and the character b repeated c times, or the number a multiplied by b to the power c.
Most people don’t worry about it, because they usually only use * and ^ for strings mixed with constant string values, i.e. "foo" * str * "bar" * 's' ^ cnt, where the meaning is obvious.

I encouraged people to always use string and repeat instead, in the case where the meaning was not clear,
i.e. string(a, repeat(b, c))

[TsurHerman]

I don’t think punning is bad, who is to say the Base is the right term and anything else a pun?
I also think that methods should be allowed to merge , reduce can mean related but different things defined in different modules , which have no interest in extending each other because they are unaware of each other.

We can qualify everything, but I prefer to just “merge” the functions and be notified if a specific use failed to resolve unambiguously.

The problem I see with extending methods in other modules , is specifically related to binary cache. I’ll explain:
Lets say you want to save all of your compiled code, to avoid re-jitting over and over again, this is crucial when developing large amounts of code…or when importing a package that represent a lot of code.

This is possible only if you can guarantee that your cache would behaves the same if read from cache and if it was just now compiled.

When you are adding a method to a module , you immediately invalidate the binary cache for that module.
If that module is Base, you would probably invalidate the entire cache.

This reason alone gives me big incentive to come up with a formality that still allows multiple dispatch which ?I love but in which it is not so easy to invalidate the entire binary cache.

binary cache: in C/C++ this would be the .o files in the build dir.

Type Piracy:
try this

import Base.+
+(a::Int64,B::Int64) = "Pirate!"

[ScottPJones]

@TsurHerman

I don’t think punning is bad, who is to say the Base is the right term and anything else a pun?

I don’t think that’s what people have been saying.
My understanding is that punning is really bad when a name has different meanings in the same namespace.

It’s not a problem that there are separate Meta.parse, Base.parse, JSON.parse, etc. (although I believe there is a better way to handle Meta.parse vs. Base.parse.)
The problem is when something has different meanings and you can’t tell what it means just by looking at the module and the code in question.

[TsurHerman]

Agreed

[cortner]

Here is an example, I just ran into. It seems to me that this is type piracy, and I should avoid it by defining a new method that is local to my module:

import Base.*
*(A::Vector{T}, b::SVector{N,T}) where {T,N} = [ a * b for a in A ]

this arose because I have a complex assembly of a vector of SVectors, but suddenly it turned out that I wanted to use the same assembly (generic programming!) with different types which required that I can perform the operation above.

I am guessing that maybe this addition to * will likely not cause bugs “right now”, but it could cause problems e.g. if somebody else has made a similar overload of * in a different module somewhere, so to future-proof this code, I should define my own method mymul. Do I understand this issue (and solution) correctly?