Is it OK to structure my packages like this (benign type piracy)?

General Usage
, , ,
1

Is this design OK? Is this supported by precompilation? What could go wrong? Could package extensions help here somehow, if necessary?

The idea is basically to define a function with no methods in PackageA and only add methods to it from PackageB. It’s type piracy, but maybe it’s OK if I document it. I think the only way this can go wrong is if someone purposefully makes a third package with the intent of breaking my packages (however that’s always possible, in any case, whether my packages do type piracy or not).

A basic example

package PackageA

Depends only on julia.

module PackageA
    """
        a::Function

    Only add methods if you're `PackageB`!
    """
    function a end
end

package PackageB

Depends on julia and on PackageA.

module PackageB
    using PackageA: PackageA

    # benign piracy?
    function PackageA.a(l, r)
        l - r
    end
end

A slightly more fleshed-out example

package PackageA

Depends only on julia.

module PackageA
    export A

    """
        a::Function

    Only add methods if you're `PackageB`!
    """
    function a end

    """
        A::Type

    Some type. Has multiplication defined on it only after loading `PackageB`.
    """
    struct A <: Number end

    function Base.:(*)(l::A, r::A)
        a(l, r)
    end
end

package PackageB

Depends on julia and on PackageA.

module PackageB
    export B

    using PackageA: PackageA, A

    struct B <: Number end

    function Base.:(*)(l::B, r::B)
        A()
    end

    # benign piracy?
    function PackageA.a(l::A, r::A)
        B()
    end
end
2 Likes
2

This is more-or-less the design chosen for ChainRulesCore.jl (PackageA) and ChainRules.jl (PackageB). ChainRulesCore.jl defines the functions frule and rrule, and ChainRules.jl defines lots of methods of them for functions )that it does not own) in Base and the standard libraries.

These were designed prior to the existence of package extensions though – a different design based on package extensions would probably be chosen today.

2 Likes
3

Type piracy means defining methods on functions you don’t own for arguments whose types you don’t own.

I would consider “you” to actually mean “you”/“your organization”, not necessarily “your particular package”. So, if you own PackageA and PackageB, then what you’re proposing should be okay (although it may not be the best design). This is probably a pretty common pattern between a user-facing package and a “base” package.

I would point out that

"""
     a::Function

Only add methods if you're `PackageB`!
"""

isn’t right: Anyone can and should add methods to a as long as they own any of the types of the arguments they want to pass to a. But you probably just mean “don’t add methods here”.

If you don’t “own” both PackageA and PackageB, then this approach is type piracy, and you should absolutely not do it. In particular, if PackageA and PackageB are not tightly coupled and coordinated, then it is the responsibility of PackageA to define methods for a for any Base types, or any types defined in PackageA.

5 Likes
4

While I do agree that this is a social aspect that is involved as well, I think that is type piracy. If you expect the code to work a certain way when just loading A this behaviour might change by just loading B with the type piracy. I think I personally would avoid that (and hopefully did until now).
Even if you own both packages, a user might get confused by this change of behaviour of “things coming (just) from A” by loading B.

5 Likes
5

The way that I interpret the manual, this is okay if PackageA and PackageB are closely coupled (which seems to be the case). The only thing that that I can see going wrong is a bit of confusion about the changing behaviour of the * operator for people that do not have the docstring fresh in their minds and some negativity towards the package for this reason.

The function a is internal and so theoretically, people should not use it directly or define even non-pirating methods for it. I guess that one other disadvantage would be that people wanting to contribute code to any of these packages might experience difficulties with this.

1 Like
6

The danger of type piracy is changing the method, or lack thereof, to which a previously possible function call dispatches; the advice about writing methods with new functions or types only concerns function calls that were not previously possible e.g. a(::A, ::A) was possible with PackageA, but a(::B) was not until PackageB. PackageC can still import PackageA and completely rely on a(::A, ::A) calls failing for some weird reason, and that’ll be broken by a PackageB import. As mentioned, that sort of usage can be discouraged by documentation or by practice, e.g. importing ChainRulesCore is intended for third parties extending (no piracy) the larger ChainRules without having to import it until it’s needed. You’re not allowing people to extend PackageA though, so I can’t think of a way this could be refactored into package extensions like ChainRulesCore could.

My objection to the idea would be that PackageA is not usable on its own or extendable, so it might as well be a submodule of PackageB so nobody can import PackageA on its own. I’m guessing you’re trying to leverage precompilation in a certain way, but it’s not clear how or why.

AFAIK precompilation only stops direct method overwriting, not all type piracy, because that causes dependency on loading order. Type piracy without method overwriting should just invalidate code possibly loaded from caches, which I would assume is irrelevant to method-less functions.

2 Likes