Name this method extension technique, piracy evasion

General Usage
,
1

Let’s say I want to change how Base.write works with a Base type, NTuple{N,UInt8} from a package.

Currently, writing NTuple via Base.write errors.

julia> iob = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)

julia> data = (0x01,0x02,0x03)
(0x01, 0x02, 0x03)

julia> Base.write(iob, data)
ERROR: MethodError: no method matching write(::IOBuffer, ::Tuple{UInt8, UInt8, UInt8})

julia> typeof(data)
Tuple{UInt8, UInt8, UInt8}

julia> typeof(data) == NTuple{3,UInt8}
true

Under type piracy rules, I should not do the following because

  1. I did not originate the types.
  2. I did not orginate the method.
function Base.write(io::IO, data::NTuple{N,T}) where {N,T} 
    sum(map(1:N) do i
        Base.write(io, data[i])
    end)
end

julia> write(iob, data)
3

To evade commiting piracy, I will originate a new method called Foo.write.

module Foo
    # Edit: comment out the export here. People should do this more explicitly.
    # export write
    # Fallback to Base implementation
    write(args...; kwargs...) =
        Base.write(args...; kwargs...)

    function write(io::IO, data::NTuple{N,T}) where {N,T} 
        sum(map(1:N) do i
            Base.write(io, data[i])
        end)
    end
end

This is not piracy because I did not violate both criteria I outlined above. I can use this as follows in a fresh REPL session.

# Explicitly indicate the use Foo.write instead of Base.write
julia> using .Foo: write

julia> @which write
Main.Foo

julia> iob = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)

julia> data = (0x01,0x02,0x03)
(0x01, 0x02, 0x03)
                                                                  
julia> write(iob, data)
3

julia> write(iob, 4)
8

julia> write(iob, 0x5)
1

julia> take!(iob)
12-element Vector{UInt8}:
 0x01
 0x02
 0x03
 0x04
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x05

One could also use a baremodule to not have Base imported.

Another important effect is that I do not invalidate any uses of Base.write and thus do not force recompilation for any other packages.

Using the language of object oriented programming, Foo.write is a submethod that inherits from Base.write. Foo.write does everything Base.write does but it also writes NTuple. Is there a preexisting name for this?

Can you think of a better name?

Edit: Commented out the export write from the example.

2 Likes
2

Well, if someone else export write, now you have conflict? Basically Plots.plot and Makie.plot

3

That is not my understanding, Foo.write calls Base.write but functions/methods do not inherit from each other. A method in a subclass may override a method in a superclass, providing single dispatch. Foo is a separate module from Base, not a subclass inheriting from a superclass.

2 Likes
4

This is distinct from that case. For plotthe underlying implementations are from completely different origins. Here there is a shared origin. The case for wanting to use both simultaneously is much smaller. Here Foo.write does everything that Base.write does, but Foo adds a capability where Base errors. If you need Foo’s extra feature, then there is no need to access the Base version.

1 Like
5

Formally, you are correct. There is no explicitly relationship between the two via the type system.

Effectively though by forwarding all arguments the Foo.write has inherited all of the functionality of Base.write

write(args...; kwargs...) =
        Base.write(args...; kwargs...)

The analogy that I making is that the relationship between Foo.write and Base.write is similar to that between subclass and its parent class. Rather here we have a submethod and a method. Changing the method will affect the submethod. Changing the submethod will not affect the parent.

Effectively, what we have in Julia is method oriented programming enabled by multiple dispatch.

I am not claiming that any relationship exists between the modules.

6

Yes, from a type theoretic perspective, you are correct (though this has nothing to do with multiple dispatch). I wouldn’t bring object orientation/classes into this (there’s way too much baggage around that term) - what you’ve found is simply a case of behavioral subtyping (which admittedly originates in object orientation).

The issue that comes with your definition of type piracy though is that because you don’t own any argument types and you intend your Foo.write to otherwise behave exactly the same as Base.write, as soon as Base.write adds a method with the same signature, your code is either broken (it doesn’t do the same thing as Base.write anymore) or you’ve violated the contract of Base.write (not extending it with methods on types you don’t own). So while you’ve technically skirted around your definition of piracy in the literal sense, I’d argue that you’re still comitting type piracy, because there’s an expectation from Base.write that it’s free to define methods for types it owns.

It gets a bit muddy mostly because you’ve called your function write too - if it were called differently, there wouldn’t be any issue, because the expectation that it would behave like Base.write wouldn’t exist.

Put differently, you’ve assigned your Foo.write a contract it can’t possibly fulfill - being exactly like Base.write, while doing some things potentially different from Base.write, all the while having no control over what Base.write may decide to (rightly so) do on types it owns.

2 Likes
7

Not really – this isn’t type piracy even when Base adds a method with that signature. You’re just shadowing the original implementation with your own, which may of course have disastrous consequences, but at least they’re local (whereas actual type piracy has global consequences).

Of course, the non-global nature of this shadowing also limits its usefulness compared to actual type piracy (well, modifications to the global method table in general).

That contract can be fulfilled at any given point of time. It just might not hold for future releases any more.

5 Likes
8

My point is that forward compatibility (with Base.write in this case) is part of the contract of any well-behaved stable API; that’s not something Foo.write can provide, because there’s no room for a transition if Base decides to make that change and the method differs from the one Foo.write does. The only option left is to drop the requirement to be equivalent to Base.write; that is, having this as the contract of Foo.write:

    Foo.write(io::IO, args...; kwargs...)

Except for `Foo.write(io::IO, NTtuple)`, `Foo.write` passes its arguments to `Base.write`.

is perfectly fine & well behaved, because it explicitly states that it doesn’t care what Base does with NTuple arguments - that difference is the defining feature of Foo.write after all. What’s left is the easily confused, but intentionally chosen name - to me an indicator that this shouldn’t be done :person_shrugging:

2 Likes
9

Fair enough. Whether the name is confusing is debatable though, because Foo.write still satisfies the “definition” (intention) of Base.write, specifically

Write the canonical binary representation of a value to the given I/O stream or file. Return the number of bytes written into the stream

So in this specific case I’d argue that it’s impossible for Base to define a method for these input arguments that results in a different output to Foo.write.

10

The behavior of Foo.write may be faithful to the intention of Base.write, but they could do it in measurably different ways. Foo.write here specializes on every tuple type, Base.write could @nospecialize instead to avoid more compilation.

11

I’m saying another package can do the same thing as your Foo.

e.g. Bar.write might get exported and now users will see conflict

1 Like
12

@jling, yes, a namespace conflict, not a method override conflict, I think it would be good to make this explicit. If someone does not employ using PackageName, and instead always import specific names this will not happen. And, in fact, any package can have this problem (i.e., any two packages can have exported functions with the same name and this may have nothing to do with piracy evasion or delegating to Base, it may be just an unfortunate coincidence).

About the taxonomy, it may be that I am being pedant, but I think the terms would be delegation or forwarding instead of inheritance. Which one of the two is a very subtle detail: Forwarding (object-oriented programming) - Wikipedia

2 Likes
13

exactly, I guess my point was that this does evade piracy, but it doesn’t actually solve underlying problem, it’s not a solution that can be adopted by ecosystem because it doesn’t scale.

If people replace their want-to piracy with exporting shadowing Base.blah(), it would be pretty annoying

14

This does scale better than piracy itself. The export declaration is probably too much of a distraction here, and people should explicitly declare they are using it. I’m commenting out the export.

I could imagine an ExtendedBase.jl that shadows all of Base but allows for faster iteration than Julia itself. It would be a forward looking version of Compat.jl in a sense.

1 Like
15

Yes, I think the main idea of this pattern is to allow for Quality of Life (QoL) improvements. Ideally, this could be a way for solve problems before Julia/Base releases a new version and then have the code adopted in Base if it seems like a good idea.

16

I like “forwarding”. That seems very intuitive. I’m forwarding all the positional and keyword arguments. I’m not sure if the difference between “delegation” and “forwarding” applies here since there is no wrapper object involved.

This issue exists in object oriented programming as well. If the parent base class changes, then the subclass will be affected.

I would document the contract as follows.

This method forwards to Base.write except for the following overloads.

  1. write(io::IO, x::NTuple{N,T}) where {N,T} which normally errors for Base.write.".

Yes. Perhaps this behavior, with the export now gone, is limited to within a module, so this could be seen as a form of private method overloading.

17

I was thinking that a syntax function write <: Base.write end might be nice. This is currently a syntax error, so perhaps this could be valid later. For now we could use a macro to do enable the following syntax: function write() <: Base.write end. This also allows for supertype(typeof(Foo.write)) to work.

module MethodForwarding
    export @methodfwd
    macro methodfwd(ex)
        if ex.head == :function && ex.args[2].args[end].head == :<:
            fn = esc(ex.args[1].args[1])
            parent = esc(ex.args[2].args[end].args[1])
        else
            error("@methodfwd can only be applied to explicit `function` declarations that containing a unary `<:`")
        end
        quote
            $fn(args...; kwargs...) = $parent(args...; kwargs...)
            Base.supertype(::Type{typeof($fn)}) = typeof($parent)
        end
    end
end

We could then write Foo as follows.

module Foo
    import ..MethodForwarding: @methodfwd
    @methodfwd function write() <: Base.write end
    function write(io::IO, data::NTuple{N,T}) where {N,T}
        sum(map(1:N) do i
            Base.write(io, data[i])
        end)
    end
end

Then do the following.

julia> supertype(typeof(Foo.write)) == typeof(Base.write)
true

julia> Foo.write == Base.write
false
18

I like the general idea of this, but only in combination with something like Taking API surface seriously: Proposing Syntax for declaring API · Issue #49973 · JuliaLang/julia · GitHub or this experiment I plan to use for PropCheck.jl (the change integrating the experiment into the latter is already done, I’ve just got to finalize & push it to github, after which I’ll register both packages).

My one counterpoint to function write <: Base.write end is that what you’re actually claiming with such a declaration is that your write is behaviorally compatible with Base.write , which it (as mentioned above) cannot be, as Base.write is (currently) free to define whatever methods on Base types it pleases. Not defining Base.write(::IO, ::NTuple) simply isn’t part of the contract of Base.write.

I’m also not sure forwarding like that is very useful in practice - what problem does it solve, other than being a way to get around type piracy?

1 Like
19
  1. I encapsulate the method overload to the current module.
  2. I avoid causing method invalidations for other packages
  3. Others do not have rewrite their code to use my modified method in their packages. They can just import the version from my package.

The alternative is that I give my method a completely new name.

module NTupleFriendlyWrites
    ntuple_friendly_write(args...; kwargs...) = Base.write(args...; kwargs...)
    function ntuple_friendly_write(io::IO, data::NTuple{N,T}) where {N,T} 
        sum(map(1:N) do i
            Base.write(io, data[i])
        end)
    end
end

I suppose people could just do const write = NTupleFriendlyWrites.ntuple_friendly_write though.

20

As pointed out above though, as soon as people actually want to use this, they’ll get namespace conflicts with Base. Not to mention that punning on the name/shadowing the definition from Base means inconsistent expectations when people don’t use the package.

I think it’s generally better to try to upstream such additions to an API, instead of wrapping & punning on the name.