Method overwriting error on package extensions

General Usage
, ,
1

Hello!
I have been trying to write two extensions for a package so that it can operate with Plots.jl or Makie.jl. Everything works fine but when I run the tests I get a series of warnings and errors that look like this:

WARNING: Method definition concordiacurve!() in module Isoplot at /.julia/dev/Isoplot/src/generic_plotting.jl:3 overwritten in module PlotsExt at /.julia/dev/Isoplot/ext/PlotsExt.jl:66.
ERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.

The tests still run and pass despite these warnings and errors. Adding
__precompile__(false) to the front of the extension modules removes the errors but the warnings remain and this is not a practice I have seen in other packages that use extensions.

It seems bizarre to me to have warnings about type piracy for package extensions, is this intended behavior or is it possible that I have missed something in implementing the extension? I am using version 1.10.3.

Thanks!

2

Probably you need to import the method into the extension, to add a method, not overwrite it. Something like:

module PlotExtension
     using MainPackage: MyType
     import Plots
     Plots.plot(x::MyType) = ...
end
3

I explicitly import the methods that I am overloading from the main package. Here is an example from the Makie extension:

module MakieExt

    using Isoplot
    using Makie
    using Measurements
    import Isoplot: concordialine, concordialine!, concordiacurve, concordiacurve!
4

I had a similar situation that I found confusing as well. To me it seems like extensions are a benign form of type piracy that should not trigger such dire warnings. (Also, when I see “ERROR”, I don’t expect that the code will then run anyway…)

I think I have gotten around it by making sure that the method signatures in the main package and the extension are not the same. For example, here is a package src file:

module dumdum

export myplot
myplot(f::Any) = error("not defined")

end # module dumdum

and then in ext I have

module DumExt

using dumdum, Plots
dumdum.myplot(x::Integer) = Plots.scatter(rand(100))

end

Then, with Project.toml aware of the extension, I can do this in v1.10.4:

julia> using dumdum
Precompiling dumdum
  1 dependency successfully precompiled in 1 seconds

julia> myplot(1)
ERROR: not defined
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] myplot(f::Int64)
   @ dumdum ~/tmp/dumdum/src/dumdum.jl:4
 [3] top-level scope
   @ REPL[2]:1

julia> using Plots 
[ Info: Precompiling DumExt [0da8e23a-6336-5d3f-a165-06475ef3b352]

julia> myplot(1)  # makes plot

Since the precompilations were silent, I assume this means they were happy. But if I change the signature in dumdum.jl to ::Integer, I get the “skipping precompilation” message when using Plots.

5

It’s not type piracy at all if you’re only affecting your extension or your package it’s attached to because you have full control over the dispatch. You can indeed cause similar negative effects if you’re not careful. Method overwriting and type piracy are separate problems, method overwriting can occur within the same module.

Then you have 2 dumdum.myplot(::Integer) methods, which precompilation caches can’t handle at once. Consider a couple things:

  • other packages that import dumdum. Which method would they use when precompiling their own code? It might be tempting to say the first one in dumdum for the more general case, but then using dumdum, Plots would switch to the useful method and make all those packages’ caches wrong. A lot of effort went into reducing wasteful compilation before precompilation could develop, so this is the wrong direction.
  • In principle, loading packages in a different order should result in the same code. Here it appears there is a simple order: extension must load after the packages. But what if you had another extension DumMakieExt with using dumdum, Makie making another myplot(::Integer) method? If you do using Makie, Plots, dumdum, which method survives? If another package imports those 3 packages, what does it precompile with?

As the developer, you can choose one of the methods to keep. This limitation of dynamism is expected from AOT compilation, and it’s not the hardest precompilation limit to deal with in Julia.

Not sure what your example is trying to do exactly, but I prefer methods in extensions to mix functions and types from different packages together in the method signature, not just in the body. It’s more fine here because myplot is supposed to be useless without the extension, so you won’t have other methods using myplot and being invalidated when it gets more specific methods. I would probably go farther and just declare function myplot end.

1 Like
6

Not sure what is it you don’t understand, the error message basically says it all. Just don’t overwrite methods across packages/extensions. EDIT: in retrospect my message is too rude. Sorry.

Just do function myplot end.

7

Something that prints WARNING and then ERROR on every invocation is enough to give anyone pause. As a long-time Julia user, I was fairly but not completely certain that I was not losing much. But to two users of my package, the message was unclear and unsettling enough for them to file an issue about it. I don’t blame them; like a lot of Julia errors and warnings, they suffer from the curse of knowledge.

The real reason I use the error stub in the main module is to print out a tip to load a Makie package and try again. This step is obvious to someone who knows how extensions work but could cause lost time or giving up to a newcomer.

8

You’re losing precompilation, as the error message says.

That’s fine and expected.

As @Benny says, a nicer and better design would be to have Makie or some type from Makie in one of the documented method signatures. Then, apart from the other benefits like extensibility, users would get an error when trying to use Makie if it’s not loaded.

9

The typical case would be to extend a plotting function from a plotting package on your package’s custom type. That would be a seemless change in a call from Plots.plot(::Vector, ::Vector) to Plots.plot(::DumVector, ::DumVector).

Not so in this case where the plotting package is not mentioned at all in the call. This resembles plotting packages having multiple backends more than typical function extension. I’m just not sure if that’s doable with package extensions. Going back to my point about how loading packages in any order should have consistent behavior instead of arbitrarily invalidating a subset, it should also be true that multiple backends shouldn’t sabotage each other, if multiple backends are even supposed to be loaded at the same time. Compare this to Plots where function calls switch backends (with some limitations); it’s not feasible for using Makie to undo an earlier using Plots, then be undone by a subsequent using Plots because a package is loaded once per process and can’t be unloaded. I don’t actually know how Plots handles its backend loading, but it’s clearer for Makie where the multiple backends are their own packages that you load directly.

10

He could dispatch on, e.g., Val(Makie).

11

He could, but it’s not a nice design for plotting. I wouldn’t mind it if I were changing backends chaotically, but typing is obnoxiously repetitive if I can stick to one backend for a while between changes, which is much more common. You could save typing with derived functors storing the backend state, but that complicates the API and risks unintended backend clashes. This is one of those rare cases for global state, which Plots and Makie jumped a lot of hoops to pull off. Wonder how they deal with multiple backends coexisting, that would be valuable insight.

12

Ok so it seems that my issue was that some of my functions in my extensions had zero arguments and the compiler had no way to resolve these with the dummy functions in the main part of the package. This was resolved by using a dummy type in the dummy functions. Example:

struct NotUsed end

function dummy_function(x::NotUsed) end
1 Like
13

@sc-dyer Pretty sure you’re still misunderstanding things. It’d help if you gave a MWE, but I think you’re implying that, in the extension,

function dummy_function end

… doesn’t work for you, while …

struct NotUsed end

function dummy_function(x::NotUsed) end

… “works”.

Two things:

  1. Why are you trying to define “dummy” functions/methods in an extension? The purpose of an extension is to add new functionality.

  2. The difference between the two code blocks above is that function dummy_function end defines a function without giving it a method, while the other form defines a function with a single method. You can see that in the REPL:

julia> function dummy_function end
dummy_function (generic function with 0 methods)

julia> struct NotUsed end

julia> function dummy_function(x::NotUsed) end
dummy_function (generic function with 1 method)

Defining “dummy” methods for your “dummy” functions really doesn’t seem like the right thing to do, whatever it is you’re trying to do.

14

My apologies, I think I wasnt clear about what I was referring to. The example I gave was in the main body of the package and then I define additional functionality in the extension.

So where before I would have in the main package this:

function concordiacurve() end

and in the Makie extension I had this:

 @recipe(ConcordiaCurve,t₀, t₁) do scene
...
end

function Makie.plot!(ccurve::ConcordiaCurve)
...
end

that would give me the warnings in the original post. But now in the main package I have this:

struct NotUsed end

function concordiacurve(x::NotUsed) end

and that makes the warnings in the original post go away. I believe this is because Makie recipes define a concordiacurve function that does not take any arguments leading to issues with a argument-less definition in the main package.

15
1 Like
16

Oh fair – I suppose I never realized function foo end was an option, but makes sense