How to extend method used in package extension

General Usage
, ,
1

Hello,

I am using a package which has plotting functionality defined in a package extension. It uses a function package_function which is defined in the package module as

function package_function end

and in the module PlotExt, there are various methods for package_function. I would like to create a method package_function(x::MyNewType). I used

import ThePackage: package_function

and defined the method for the new type. It is visible in main, but is not visible by PlotExt. Is there a way to extend package_function so that it is visible by PlotExt?

2

I’m slightly confused where import ThePackage: package_function is. If you put it in PlotExt, then it should be available to extend there.

module PlotExt
    # ...
    import ThePackage: package_function
    # ....
end

The problem you might run into is if MyNewType is available and exported.

If you could be more concrete on how you have arranged your code and where your import statements are, that would be helpful to give you a specific answer.

3

@mkitti, thank you for your reply. Let me see if I can clarify. I am using a package developed by someone else. So I do not want to include MyNewType and package_function(x::MyNewType) in that person’s package. I would like to define those separately in my own project:

my_script.jl

using Plots
using ThePackage
import ThePackage: package_function 

struct MyNewType
...
end

function package_function(x::MyNewType)
...
end 

x = MyNewType()

# this function is defined in PlotsExt of ThePackage
# it calls package_function
outer_package_function(x)

When I run this code, I receive an error originating from PlotsExt indicating that package_function(x::MyNewType) is not defined. In the past, I have used a similar pattern for code defined within the ThePackage module. However, it does not seem to work for PlotExt.

4

Can you provide the error’s full stacktrace and check ThePackage.package_function === Base.get_extension(ThePackage, :PlotExt).package_function? Or :PlotsExt, however it’s really spelled (you wrote both so far).

5

What are you describing is not a package extension. A package extension is a glue package that loads when two or more other packages are loaded.

Instead you are actually adding new methods and new types. In this case, this should just be a regular package that depends on the other package.

You should not insert MyNewType or package_function into ThePackage’s namespace. Instead, your package could reexport all of ThePackage’s symbols adding the new type and new functions to it. There is a package Reexport.jl that assists with this pattern.

module ThePackage
    export foo, Bar
    foo() = println("Hello")
    struct Bar
    end
end

module MyPackage
    using Reexport
    @reexport using ThePackage
    export MyNewType, my_function

    struct MyNewType
    end
    function my_function(x::MyNewType)
    end
end

Technically it is possible to insert methods and functions into foreign packages via @eval but this is highly discouraged and breaks a lot of things such as precompilation caches. For example, the following is how one would insert a type into the Base module.

julia> Base.Foo
ERROR: UndefVarError: `Foo` not defined

julia> struct Base.Foo
           x::Int
       end
ERROR: syntax: invalid type signature around REPL[2]:1

julia> @eval Base begin # Do not do this
           struct Foo
               x::Int
           end
       end

julia> Base.Foo
Base.Foo

julia> Base.Foo(5)
Base.Foo(5)

Here’s a fully self contained in-REPL demonstration. The main difference here from a real scenario is that the packages are defined inline as Main.ThePackage and Main.MyPackage, so we have to refer to them relatively.

julia> using Pkg

julia> Pkg.activate(; temp = true)
  Activating new project at `/var/folders/5j/phv_vfc97m967ww98fbvtmfr0000gq/T/jl_k1C41J`

julia> Pkg.add("Reexport")
   Resolving package versions...
    Updating `/private/var/folders/5j/phv_vfc97m967ww98fbvtmfr0000gq/T/jl_k1C41J/Project.toml`
  [189a3867] + Reexport v1.2.2
    Updating `/private/var/folders/5j/phv_vfc97m967ww98fbvtmfr0000gq/T/jl_k1C41J/Manifest.toml`
  [189a3867] + Reexport v1.2.2

julia> module ThePackage
           export foo, Bar
           foo() = println("Hello")
           struct Bar
           end
       end
Main.ThePackage

julia> module MyPackage
           using Reexport
           @reexport using ..ThePackage
           export MyNewType, my_function
           
           struct MyNewType
           end
           function my_function(x::MyNewType)
           end
       end
Main.MyPackage

julia> using .MyPackage

julia> foo
foo (generic function with 1 method)

julia> Bar
Bar

julia> MyNewType
MyNewType

julia> my_function
my_function (generic function with 1 method)

julia> foo()
Hello

julia> @which foo
Main.ThePackage

julia> @which Bar
Main.ThePackage

julia> @which MyNewType
Main.MyPackage

julia> @which my_function
Main.MyPackage
6

I was assuming that there was an unposted package extension and that outer_package_function really originated in ThePackage and is only implemented further in PlotExt, but this is making me question things too.

How is PlotExt structured, and is it in the /ext directory of ThePackage?

7

I apologize for the confusion. I think Benny is correct about what I am trying to do. It turns out that I was mistaken about where package_function was defined. It was only defined in the PlotExt module. When I dev’ed the package and added function package_function end to the main package, and then imported it into PlotExt, I was able to extend the method for the new type and it worked with PlotExt. Sorry again for the oversight. I am still getting familiar with the package extensions.

8

The key concept there is that unlike packages, extensions and the names within them are not importable. So, their purpose is mixing names from their package dependencies, though it is allowed to create internal names to implement things. Do not be misled by Base.get_extension; while that lets you access an extension’s module, assigning anything from it to a new variable is not equivalent to importing (and optionally renaming) an existing variable. For now, that seems more useful for code reflection, and names should strictly be imported from packages; for example, instead of an extension, make a package with the same dependencies and import that in addition to them.

1 Like
9

I’m still not following why we need this to be a package extension. You do not own ThePackage, so why are you building an extension? Why not just create MyPackage?

An extension to MyPackage would only be useful if ThePackage is an optional (weak) dependency of MyPackage and you needed to add some functionality to integrate MyPackage and ThePackage together.

10

@mkitti sorry. I think I am having trouble communicating the issue. Let me try to clarify again with an extended example:

code for someone’s package

module SomePersonsPackage 

    # the problem was that some_function was not defined in the package 
    function some_function end 
    ...
end

module PlotExtensionForSomePersons 
    
    import SomePersonsPackage: some_function

    function outer_function(x)
        
        # I want to use my own method here
        some_function(x)
        ...
        # I want to use the existing functions here
        # so I can re-use existing code
        cool_function_I_want_to_use(x)
        ...
    end
end

my_script.jl

using Plots
using SomePersonsPackage
import SomePersonsPackage: some_function 

struct MyNewType
...
end

function some_function(x::MyNewType)
...
end 

x = MyNewType()

outer_function(x)

The structure of the example above is correct. I want to use all of the functions defined by SomePersonsPackage, except some_function. The default behavior of some_function does not work for my use case; so I defined my own method for some_function to work with MyNewType.

The problem—which I did not realize initially—was that some_function was not defined in SomePersonsPackage. I confused it with a different function which was defined in both places. So extending some_function will work so long as it is also defined in SomePersonsPackage and imported into the plotting extension.

I hope that clarifies the issue and the eventual solution. If not, no worries. I eventually figured out my silly mistake. I want to thank you for your assistance!

11

I think you are referring to the term “package extension” without realizing that this refers to very specific concept.

https://pkgdocs.julialang.org/v1/creating-packages/#Conditional-loading-of-code-in-packages-(Extensions)

12

I’m not sure this is the case. It reads to me like he added two packages he does not own, ThePackage and Plots, and ThePackage has an extension PlotExt dependent on Plots. He wanted to add a method to package_function in his own script, but it only existed in the extension, not either of the packages, so it couldn’t be imported to begin with; import ThePackage: package_function should’ve printed a warning saying so, and the script’s package_function definition would’ve made a separate new function. deving ThePackage and moving package_function’s origin to the package made it possible.

Seems to me that the owner of ThePackage never intended for outer_function or some_function to be exposed. I have no idea what it’s for because it doesn’t obviously extend Plots and ThePackage, maybe it’s some internal detail ran at the top level during precompilation or in __init__() at loading.

1 Like