How to detect/avoid type piracy?

[Barget]

Thanks for bringing the distinction piracy/punning to the front, this helps clarify the discussion.

I think a the core, the difference of views comes from:

@vvbond

I do not disagree that everything acquires meaning through its context. IMHO, it’s a rather trivial statement and as such of little practical use.

I’m afraid by overviewing this aspect, you miss the point I tried to convey (in a clumsy way, possibly).

When you look at it, there’s only one place where a function gets defined, which is within its parent module.
On the contrary, methods - whether “original” or by punning - can be defined in the parent module or in some other custom modules respectively.

Thus, the intent of the function is quite univocal, and is only up to the writer of the parent (original) module. All other punning methods are strongly compelled to (or should I say requested to) comply to the original purpose, in a similar fashion to interfaces. In that view, this is: “deviate at your own risk”.

… at least, this is the consequence of the interpretation framework I tried to convey.

The practical use is then quite straightforward: look at the function to understand what you can/can’t do when punning without messing things around. Use it as a tool for good practice (we often neglect all the good that can bring mundane good practices)

In that light, because Base.:+ is used consistently throughout Base.:+, you get this nice side effect that other functions relying on it such as sum or += works. (and this is a good thing)

---

Now to some caveats & comments:

• There’s other valid way to interpreted the possibilities offered by multiple dispatch, feel free to use your own (obviously! )
• One could be concerned that documentation is not always well and good, so the function’s purpose may be unclear. But:
  • Punning is discouraged for non-public functions (or even just using non-public functions)
  • Public functions are more often than not well documented (especially for mature packages not expected to break things)

Regarding the issue of coordination:

• There’s no need for strong coordination: just adapt to what the function is … which is not expected to change much for a package which is >= 1.0.0
• … If it is < 1.0.0, then it is at your own risk.

Lastly, regarding the example of process, I’d say that because it is very abstract, you should refrain to specialize it on custom types (there’s no need to). See Base.map or Base.reduce for example, which only specializes on iterator type (collectionor AbstractArray).
On the contrary, Base.size’s purpose is quite clear. As per help: “Return a tuple containing the dimensions of A”. Deviating from that is a good way to mess with other people’s mind.
But you’re free to define Clothing.size and return what you want with that.

Btw, this is for this very reason that you can’t import say Clothing.size into Main as size, even though the methods you’ve written don’t overlap with Base.size: otherwise, which size do you intent to use? (in e.g. Main)

[Benny]

I operate under 2, and I’m not a big fan of accidentally implementing some caller functions that never anticipated excluding your custom type via dispatch, and punning by extension. It does get a pass from me if the types are so different you would never attempt to use those caller functions, e.g. * for string concatenation versus multiplication, but I would still prefer separate names in general.

@vvbond

So, no surprise that those operations become valid upon extending Base.+(::S,::S).
Indeed, it is expected and useful.

I disagree, see what happens when you go beyond 2 elements:

julia> sum(fill(S(), 2))
true

julia> sum(fill(S(), 3))
ERROR: MethodError: no method matching +(::Bool, ::S)

[sadish-d]

@Barget says “the intent of the function is quite univocal”. But + is just a symbol. It became a function and an operator when Base made it one. Yes context is important and within Base and in conjunction with Base’s types + is a function and an operator. But it has no meaning in relation to MyModule.S. S() + S() does not mean anything until I give it meaning.

In order to give meaning to + and use the syntax S() + S(), I have to add the method Base.:(+)(::S, ::S). There is no other way. I can define :(+)(::S, ::S) inside MyModule, but I cannot use it as an operator-- only a function. I can write +(S(), S()) but not S() + S(). And the moment some other module that defines + gets brought into the namespace, it has to be MyModule.+(S(), S()). It’s the same with unicode infix operators. None of that clunkiness is necessary. I can just define Base.:(+)(::S, ::S). I don’t have to worry about what other modules are doing with the + symbol because I haven’t committed any type piracy.

The catch is, Base.:(+)(::S, ::S) might “activate” other Base functions on S. This is desirable in some contexts but not others.

Also, I don’t think we can rely on the fact that defining Base.+ for our own types makes other Base functions available unless explicit promises have been made. And maybe there are such promises made for +, I don’t know. Otherwise, if Julia developers decide to reimplement sum without using +, it would break things.

[abraemer]

@sadish-d

I can define :(+)(::S, ::S) inside MyModule, but I cannot use it as an operator-- only a function. I can write +(S(), S()) but not S() + S()

I don’t get what you write here. You can use certain unicode letters as operators just fine. Sometimes I define e.g.

⊗ = LinearAlgebra.kron
vec1 ⊗ vec2 # works just fine

So you could just use another symbol that works as an operator for your operation, e.g. ⊕

[sadish-d]

module MyModule
struct S end
+(::S, ::S) = true
export S
export +
end
using .MyModule
# can't use `+` without qualification because `Base` has one too
S() + S() # WARNING: both MyModule and Base export "+"; uses of it in module Main must be qualified, ERROR: UndefVarError: `+` not defined
# can't use + as an operator even when qualifying the module
S() MyModule.+ S() # ERROR: ParseError:
MyModule.:(+)(S(), S()) # can use it as a function

[abraemer]

@sadish-d

module MyModule
struct S end
+(::S, ::S) = true
export S
export +

Yes with + this does not work because + is already defined (see warning).

Just use another symbol:

julia> module MyModule
       struct S end
       ⊕(::S, ::S) = true
       export S
       export ⊕
       end
Main.MyModule

julia> using .MyModule
julia> S() ⊕ S()
true

[Barget]

I was going to answer as Abraemer. So moving on to other points:

The other thing is, once a method is specialized (by punning) with your type, why would “activating” other functions using this method would be an issue?
At least in the case of “+”, this is definitely not.

I feel such general function with side effect issues (in the frame of “punning”) is quit rare.

Last point, the “+” has not been chosen for eg string concatenation for the reason that it suggests commutativity. Hence the “*” instead. (There may be other reasons.)
This looks quite consistent to me with the general meaning of “+”.

I’m afraid I may lack of imagination to see a concrete detrimental case.
Conversely, this may be because there’s no real issue in practice.

To then answer the initial question, type punning looks quite fine so no need to alert the user in anything. On the other side, given the issues with type piracy, having the package manager raising a warning that the package you’re using is doing type piracy would be desirable (and I don’t see why this couldn’t be done in practice)

[sadish-d]

Edit: Okay, so not everything I said here is true. See @mikmoore’s correction below.

@amraemer, this is what I mean. Imagine your package offers a ⊕ operator. But if the user brings in another package that exports it too, all places where your user was using ⊕ now become ambiguous. If you define Base.<operator>, you’re safe:

module MyModule
struct S end
Base.:(+)(::S, ::S) = true
⊕(::S, ::S) = true
export S, +, ⊕
end
module SomeOtherModule
struct T end
Base.:(+)(::T, ::T) = true
⊕(::T, ::T) = true
export T, +, ⊕
end
using .MyModule
using .SomeOtherModule
@assert S() + S()
S() ⊕ S() # get warning
# WARNING: both SomeOtherModule and MyModule export "⊕"; uses of it in module Main must be qualified
# ERROR: UndefVarError: `⊕` not defined

@Barget, I also cannot think of a concrete detremental case either, but I can imagine people going ahead and using methods like sum when the package author never meant to support it. They would not get MethodErrors.

[mikmoore]

@sadish-d

But if the user brings in another package that exports it too, all places where your user was using ⊕ now become ambiguous. If you define Base.<operator>, you’re safe:

On the contrary, this is when you’re imperiled. One package is at risk of pirating the other. If that other package accidentally collides with yours (at development time, or worse in some future update you didn’t anticipate) then it isn’t clear which function you’ll actually call (without careful inspection that defeats the whole convenience).

@sadish-d

In order to give meaning to + and use the syntax S() + S(), I have to add the method Base.:(+)(::S, ::S). There is no other way. I can define :(+)(::S, ::S) inside MyModule, but I cannot use it as an operator-- only a function

In a fresh REPL (where you have not used + previously):

julia> module FunkyAdd
               +(a, b) = Base.:*(a, b)
       end
Main.FunkyAdd

julia> using .FunkyAdd: +

julia> @which +
Main.FunkyAdd

julia> 3 + 4
12

Or, without the other module, in another fresh REPL

julia> +(a, b) = Base.:*(a, b) # define Main.:+
+ (generic function with 1 method)

julia> @which +
Main

julia> 3 + 4
12

Normally, one would expect @which + to indicate Base.

The reason you can’t do these definitions “normally” is because you already used Base.:+ in your module (probably just as +). Note that the REPL lives in its own module called Main. So when you say "now using .FunkyAdd: + it won’t know which of the two + functions you want in any particular context. But if FunkyAdd.:+ is the only thing ever used, it will use that (and instead Base.:+ will need to be qualified).

[sadish-d]

I didn’t know that. Thanks for correcting me.

module MyModule
using Base: +
struct S end
Base.:+(::S, ::S) = true
export S, +
end
module SomeOtherModule
struct T end
+(::T, ::T) = false
export T, +
end
# using .SomeOtherModule # assertion fails if you import this first
using .MyModule
@assert S() + S()

Is there no solution to this madness?

[pdeffebach]

I don’t get that the assertion fails. I get that + is not defined, which is the correct error. Since both modules export +, there is a conflict.

[mbauman]

You either have both modules use/extend/re-export Base.+ — in which case all three modules are re-exporting the exact same name for the exact same function, which is just fine — or you need to explicitly say which + function you want to use with a using .MyModule: + or somesuch.

[sadish-d]

Right. If you run it as a script I guess it fails before the assertion.

[vvbond]

julia> sum(fill(S(), 2))
true

julia> sum(fill(S(), 3))
ERROR: MethodError: no method matching +(::Bool, ::S)

Hm, I see your objection, which is evident in this particular case.
But is it to say that extending Base.:+ for custom types should be refrained from?

[mbauman]

@vvbond

But is it to say that extending Base.:+ for custom types should be refrained from?

No, it is exactly what you should do if you are actually doing addition. That’s the whole power of multiple dispatch.

This toy example isn’t really implementing an addition of any sort.

[PetrKryslUCSD]

It seems to me a name is only skin-deep. The same names may have multiple meanings. Then distinguishing the meaning by using a different name is natural, IMO. (Alas, natural languages do not bother to do that, referring to context to disambiguate.)

[sadish-d]

I think “overload Base.:+ if you want to define addition for your type” is potentially problematic. It’s not clear what “addition” means here. Must addition be associative (and I mean associative and not commutative this time)? Some people may say well of course. But then floating point addition isn’t associative and yet we let that slide.

I think such “interfaces”, if that’s the right word, are better defined for Arrays, Broadcasting, instance.property syntax and such things. (see: Interfaces · The Julia Language ) Are there clear promises for things like + where we say “Do X and you’ll get Y in return”?

[nsajko]

@sadish-d

Are there clear promises for things like + where we say “Do X and you’ll get Y in return”?

No, but two wrongs don’t make a right. If it’s clear an operation would have much different semantics that those existing for + in the ecosystem, try not to overload +.