Clarification on type piracy

General Usage
, ,
1

In the style guide, it talks about type piracy [1], the practice of extending or redefining methods on types you didn’t define. We’ve been told in the past not to do this, especially with methods and types defined by Base, as it can subtly break Julia.

However, we are in a situation where we really want to do this. The issue is with the Julia definition of the functions exp, log, inv, sqrt, sin, cos, tan (and other trig functions) and also det (for unrelated reasons).

We are writing an abstract algebra package which defines generic power series over any ring. In order for such power series functions to work, we must define all the functions above for the ring over which the power series are defined. This is ordinarily not a problem because we are defining them for types that we have defined.

One significant problem comes if we want to do power series over Julia types belonging to Integer or Rational. Here Julia defaults to the numerical standard of returning the result in the archimedean completion, instead of returning values in the original ring itself, as an algebraist would (if an algebraist wanted the value over a completion, they would specify which completion!). This implies that one simply cannot define power series over Julia Integer or Rational types in a way that is useful to algebraists.

However, there is a solution to this. Within our module (Nemo), we can define all these functions ourselves, with the usual algebraists definition, without first importing them from Base. Anyone who uses our module will have to disambiguate, but this seems to be a small price to pay. Nothing can break, as Julia itself does not use Nemo.

But we want to check with the community whether this is acceptable, or whether it would be considered type piracy? Just to be clear, we are asking whether we can define Nemo.exp, Nemo.inv, Nemo.sqrt, Nemo.det, on Julia Integer and Rational types (without importing from Base of course).

(Aside: we have long had a similar issue with / for division of integers, but we’ve worked around this by defining a function divexact, which we’ve gotten used to over time. It seems there is no similar workaround for /, as operators are handled separately. It wouldn’t be practical for anyone to have to disambiguate the use of operators. It wasn’t such a big deal, since we had to define numerous kinds of division anyway: Euclidean division, exact division, creation of elements in the fraction field, etc. The only thing it really ruins is copy and pasting of pretty printed output back into the REPL.)

[1] https://docs.julialang.org/en/stable/manual/style-guide.html#Avoid-type-piracy-1

3 Likes
2

Type piracy only applies to extending methods you did not define to operate on types that you also did not define. For example

module Foo
foo(::Integer) = 1
end

Foo.foo(1) # 1

module Bar
import Foo.foo
foo(x::Int) = 2
end

Foo.foo(1) # 2

The key part of this example is that Bar was adding methods to Foo.foo instead of Bar.foo. This allowed Bar to change the behavior of calling Foo.foo(1). If I was relying on Foo.foo to return 1, then defining Bar just introduced a bug into my code. However, if you delete the import Foo.foo line, you no longer have a problem.

Similarly, Nemo can safely add Integer or Rational methods to Nemo.exp if you do not import Base.exp. As long as you do this, somebody calling exp(1) won’t suddenly have the behavior of the program change after using Nemo.

2 Likes
3

Having local implementations is not type piracy, see the “extending or redefining” part.

4

I should add that we would be exporting these functions. Certainly not import, just export. Still ok?

5

Yes, it is fine, but what is the gain of exporting them?

6

There might be a way around it in some cases, but for example, we have to export exp from Nemo, because exp is a function defined on power series, not just on the rings over which you can take power series. Also, sqrt returning a float just isn’t much use to a user of Nemo. They will want Nemo.sqrt to return an exact value in the ring (if it exists), etc.

7

Nothing of what you said explains why you need to export e.g. exp. The exp binding is already available by default since it is exported from Main.

8

You are right. I think you just revealed a misunderstanding I have about what export actually does.

9

Actually, on second thoughts, Julia has changed the name of so many of its functions so many times, I think we ought not rely on a “binding” for exp being available because of an export from Main. And who knows, maybe one day they get moved into a package, as happened with isprime, for example (this was a good thing for us, I’m not criticising).

10

It still doesn’t matter because your users will qualify with module name.

11

Technically, yes. But for new users, we essentially rely on the ambiguity message emitted by Julia to teach new users that. Should Julia change the name of its function, users would have no idea why the function doesn’t work. I still think it is better to export it, and I don’t see any down side.

12

It’s a little hard to tell what you’re doing, but I have a feeling that this might not be the right (or at least not the most julian) way of accomplishing your task. Do you really need a new function called sqrt which is not just another method of Base.sqrt? That is, can you not just do:

module MyModule

struct MyNumber <: Integer
  x::Int 
end

import Base.sin

sin(y::MyNumber) = 2 * y.x

end

?

13

For the first few years we did precisely this. But now our module is more widely used, people want to do things like define power series for Julia Integers and Rationals using our module. We recently made this possible through some (perfectly Julian) trickery, which gets around the fact that Julia integers and rationals are not part of our abstract type hierarchy.

Also, we found a bunch of bugs in our code when we did this, because it allowed us to generate more exotic test examples which are more likely to trigger bugs.

Also, we are now planning to split Nemo into two packages, one of which is pure Julia, that can be used by other people without the C library dependencies that Nemo has. We considered wrapping the Julia types, but after reading answers on stackexchange on how to handle this, we decided this is not the best solution.

14

It seems like a bad API to rely on users reading and learning from error messages.

The problem is that it is not enough for a user to use Nemo.exp, but uses of exp will also need qualification with Base.exp.

1 Like
15

There isn’t any risk of an algebraist wanting Base.exp on a regular basis.

The only other solution is to use different names for power series functions, e.g. exponential, logarithm, square_root, inverse, sine, cosine, tangent, etc. However, so far, our users have absolutely refused to allow that change. They want the same names as Julia uses. So we have some constraints on possible solutions to this.

Either we don’t allow power series over Julia types, which kind of makes Nemo useless as a standalone Julia module, or we rename all our functions to something that isn’t very convenient to use or remember.

If there was another workable solution, I’d do it.

16

Well, who are you to decide which functions people are allowed to use?

Since you already require Nemo.exp the solution is to document that the way to use Nemo is to qualify functions with Nemo.

See above.

17

Could you avoid exporting by wrapping all the code scoped to use Nemo symbols …etc… in a @nemo macro that prepends all the symbols with Nemo. ?

18

I’m trying to respond to two competing demands:

  1. Allow Nemo to work as a standalone Julia module which applies to Julia types

  2. Keeping the existing Julia function names for exp, sqrt, inv, det, etc.

Julia makes this impossible by insisting on a particular definition of math functions that doesn’t make any sense in the context of a generic algebra package. It should be necessary in Julia to import math (or using Math) similar to Python. But that isn’t going to happen, so whatever we do, we are going to require some kind of workaround. Those are just the givens.

This is not necessary. It’s baked into the language that if you want to call a function provided by a given module, and it conflicts with someone else’s module, you have to disambiguate. We don’t need to explain that to Julia users. We may put a note in the manual about the functions Julia has already defined, but our experience is users don’t read the manual for every function they use. They try a bunch of things first. In this case, Julia will tell them what they need to do. At that point, if they choose to look up the Nemo manual, they will find the information they need.

The solution suggested earlier in the thread does not address 1 above. Moreover, we explicitly looked for answers to people who had asked how to handle this in the past, and they were told to use a Union of Julia types with their own types to solve this issue. This is what we’ve done and we wouldn’t look back. Clearly requiring Nemo users to have a special integer type isn’t a realistic solution.

19

Serious question: I actually don’t understand why people would want us to NOT export our functions. Doesn’t it ordinarily make the package really difficult and annoying to use if you don’t export the functions?

Yes, we could implement some macro for our users that would prepend Nemo. But what would be the motivation to do this? I don’t understand why there would be a reluctance to export functions from a module in the first place. Isn’t it standard practice?

I feel like I am fundamentally misunderstanding what export is for!

20

From your top post:

It seems like you require users to fully qualify function calls? E.g. Nemo.exp etc. So with this in mind there is 0 use in also exporting the conflicting names. The only extra “gain” if you export is that you break code for people that use exp from Base.

See above. The point is that using Nemo shouldn’t affect how you have to use other libraries (in this case Base).