# Are extension packages importable?

**URL:** <https://discourse.julialang.org/t/are-extension-packages-importable/92527>\
**Category:** General Usage\
**Tags:** package-extensions\
**Created:** [January 4, 2023, 9:15pm UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527 "2023-01-04T21:15:14Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [January 4, 2023, 9:15pm UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/1 "2023-01-04T21:15:14Z")

</div>

The new extension packages functionality looks great! [Code Loading · The Julia Language](https://docs.julialang.org/en/v1.10-dev/manual/code-loading/#man-extensions)

I get that its intended use is to provide Requires.jl-like functionality, i.e. to run some code when some other package is loaded, e.g. in order to add compatibility methods.

Two questions:

1. Is there a mechanism to import the module defined in an extension package?
2. Is there a mechanism to install the packages for an extension? E.g. if package Foo had an extension FooExt depending on ExtPkg say, can I specify to install FooExt somehow (which implicitly installs ExtPkg) instead of specifying ExtPkg explicitly?

If not, are these on the roadmap?

The second suggestion is a lot like Python’s “extras”, which you’d install like

```julia
pip install 'Foo[FooExt]'

```

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [January 4, 2023, 10:27pm UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/2 "2023-01-04T22:27:03Z")

</div>

~~My vague understanding is that extension packages are true packages on their own. You could probably add them as subdirectory packages.~~

I think @kristoffer.carlsson would be the one most qualified to clarify this.

Here is the pull request to Julia:

> <https://github.com/JuliaLang/julia/pull/47695>
>
> This is an implementation of the proposal in https://github.com/JuliaLang/julia/…issues/43119 (with some tweaks) that aims to replace Requires.jl for loading code based on the presence of other packages. It is a different implementation to https://github.com/JuliaLang/julia/pull/47040 and should work better in for example big environments where you expect to only load a small subset of packages.
> 
> In spirit, it is very similar to Requires.jl in that given a set of packages and a file, you load the file when those packages have been loaded in the julia session. The differences to Requires.jl are:
> 
> \- The file with the conditional code gets precompiled in the same way as a normal package would. Requires.jl instead directly evaluates the code into the package.
> \- It is possible to use Pkg to set compatibility on the packages that "activates" the conditional code. This is a new feature in Pkg and could in theory also be used together with Requires.jl.
> \- It is "declarative" in the sense that all information requires id stored in the Project.toml and Manifest.toml file so it avoids having to run any arbitrary Julia code in e.g. \`\_\_init\_\_\` (which is done when using Requries.jl).
> 
> The added docs in code loading as well as the added docs in the accompanying Pkg PR (https://github.com/JuliaLang/Pkg.jl/pull/3264) can be used to get a more detail description of its usage.
> 
> A lot of the diff in this PR is due to the code loading being pretty awkward and would have been easier to be built on top of https://github.com/JuliaLang/julia/pull/46690 but I don't want to couple those PRs.
> 
> For reviewers: going through all the uuid lookups etc in the code loading is probably fairly uninteresting since it is mostly mechanical, the interesting parts are those around line 1080 - 1200 in \`loading.jl\` which deals with the insertion of gluepkgs "callbacks" and the loading of them.
> 
> An example of a real package moving from Requires.jl to using this system can be seen in https://github.com/KristofferC/PGFPlotsX.jl/compare/kc/glue. It should be pretty easy to be backwards compatible with Requires. You just put the \`@require\` inside a version check and include the file for the conditional file in the \`@require\` block.
> 
> One TODO that is left is that Pkg should precompile these "glue packages" during \`Pkg.precompile\` (which is often run automatically) in the case of them having a chance to be loaded in the current environment.

and the corresponding pull request to Pkg.jl:

> <https://github.com/JuliaLang/Pkg.jl/pull/3264>
>
> Requires https://github.com/JuliaLang/julia/pull/47695

Here are some example extension modules:

> <https://github.com/KristofferC/PGFPlotsX.jl/tree/master/ext>
>
> //github.com/KristofferC/PGFPlotsX.jl/tree/master/ext

It appears that these modules do not have UUIDs so they may not be bonafide packages in themselves. The example prototypes show the extensions are usually modules which are included as submodules of the base package.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [January 5, 2023, 11:51am UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/3 "2023-01-05T11:51:14Z")

</div>

> [@cjdoris](#):
>
> Is there a mechanism to import the module defined in an extension package?

You can get the module using `get_extension`. Using PGFPlotsX as an example:

```julia
julia> using PGFPlotsX

julia> Base.get_extension(PGFPlotsX, :ColorsExt)

julia> using Colors

julia> Base.get_extension(PGFPlotsX, :ColorsExt)
ColorsExt

```

So before the extension is loaded `get_extension` will return `nothing`.

In general though, you should probably try to make getting the module not required to use the package. It is better to try have the extension add methods via dispatch and make the new functionality available in that way.

* * *

> [@mkitti](#):
>
> It appears that these modules do not have UUIDs so they may not be bonafide packages in themselves.

They get their UUID by combining the UUID of the parent package and the name of the extension module:

```julia
# Get UUID of extension module
julia> Base.PkgId(Base.get_extension(PGFPlotsX, :ColorsExt)).uuid
UUID("283d1826-985b-5544-82b8-7fd9aa83b823")

# Which is computed like this:
julia> Base.uuid5(Base.identify_package("PGFPlotsX").uuid, "ColorsExt")
UUID("283d1826-985b-5544-82b8-7fd9aa83b823")

```

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [January 6, 2023, 12:49pm UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/4 "2023-01-06T12:49:44Z")

</div>

Thank you for the replies!

> [@kristoffer.carlsson](#):
>
> In general though, you should probably try to make getting the module not required to use the package.

I get your point for the intended usage of extensions, I was more wondering if the same mechanism could be co-opted to provide something a bit like Python’s “extras”.

In a way I’m asking if it’s possible to have multiple modules in a single Julia package which can be independently imported and have separate dependencies. The answer to that is seemingly currently “no” - which is fine because you can always create multiple packages instead.

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [January 6, 2023, 12:56pm UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/5 "2023-01-06T12:56:46Z")

</div>

To be concrete, suppose the package `Foo` has an extension `Ext` which depends on `Bar` and `Baz`.

Then it may be nice to let

```julia
pkg> add Foo[Ext]

```

be a shorthand for

```julia
pkg> add Foo, Bar, Baz

```

(thus allowing the fact that `Ext` depends on `Bar` and `Baz` to be an implementation detail).

And further, it may be nice to let

```julia
import Foo[Ext]

```

be a shorthand for

```julia
import Foo, Bar, Baz
const Ext = Base.get_extension(Foo, :Ext)

```

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [January 6, 2023, 7:27pm UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/6 "2023-01-06T19:27:32Z")

</div>

It still isn’t clear to me if the name of the extension should even be something a user should ever know about.

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [January 8, 2023, 11:36am UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/7 "2023-01-08T11:36:05Z")

</div>

I guess like everything else, it’s up to the package author whether or not a particular extension is part of the API.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [January 8, 2023, 5:26pm UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/8 "2023-01-08T17:26:39Z")

</div>

Well yes, but I do think that best practices should be to make the functionality exposed by overloading functions in the parent. It can even be a zero method function that is overloaded. So you would not need to access the namespace of the extension.

---

<div class="post-metadata">

**Author:** ![t-bltg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/t-bltg/32/25526_2.png) [@t-bltg](https://discourse.julialang.org/u/t-bltg)\
**Post date:** [February 4, 2023, 10:35pm UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/9 "2023-02-04T22:35:08Z")

</div>

> [@kristoffer.carlsson](#):
>
> You can get the module using `get_extension`. Using PGFPlotsX as an example:

@kristoffer.carlsson, this is weird, I cannot get `Base.get_extension` to work in my REPL (but the tests are passing 😕 ):

```julia
julia> using PGFPlotsX

julia> Base.get_extension(PGFPlotsX, :ColorsExt)

julia> using Colors

julia> Base.get_extension(PGFPlotsX, :ColorsExt) # what ?

julia> VERSION
v"1.9.0-beta3"

julia> "PGFPlotsX" ∈ map(x -> x.name, collect(keys(Base.loaded_modules)))
true

julia> "Colors" ∈ map(x -> x.name, collect(keys(Base.loaded_modules)))
true

julia> "ColorsExt" ∈ map(x -> x.name, collect(keys(Base.loaded_modules))) # what ?
false

```

What could have gone wrong here ?

(Occurred while working on [this PR](https://github.com/JuliaPlots/UnicodePlots.jl/pull/332) , CI is OK, but I cannot get it to work in the REPL locally) …

EDIT: hum, appears to be a duplicate of [weakdeps don't load from fallback environment · Issue #48351 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/48351), and should be resolved in next beta by [allow extensions to be loaded from non top level env by KristofferC · Pull Request #48352 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/48352).

---

<div class="post-metadata">

**Author:** ![hhaensel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hhaensel/32/1207_2.png) [@hhaensel](https://discourse.julialang.org/u/hhaensel)\
**Post date:** [April 18, 2023, 7:36am UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/10 "2023-04-18T07:36:48Z")

</div>

I’d like to come back to question 1 of @cjdoris

> [@cjdoris](#):
>
> 1. Is there a mechanism to import the module defined in an extension package?

I’m not sure whether this is answered in the above posts. I understand the question is about whether there is a possibility to export/import variables from the extension module.

In `Reqires.jl` it is possible to export variables. They are part of the parent module. For the extension module I have not been able to export or import variables, neither as part of the extension module, nor via `@eval <parentmodule> export x, y` as this would break precompilation.

Am I missing something?

---

<div class="post-metadata">

**Author:** ![AbhimanyuAryan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/abhimanyuaryan/32/15733_2.png) [@AbhimanyuAryan](https://discourse.julialang.org/u/AbhimanyuAryan)\
**Post date:** [April 20, 2023, 5:09am UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/12 "2023-04-20T05:09:03Z")

</div>

same question as @hhaensel

---

<div class="post-metadata">

**Author:** ![Norman](https://avatars.discourse-cdn.com/v4/letter/n/97f17d/32.png) [@Norman](https://discourse.julialang.org/u/Norman)\
**Post date:** [April 20, 2023, 6:01am UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/13 "2023-04-20T06:01:49Z")

</div>

From what I understand, it is not considered to be a good practice to allow additional objects to be exported conditionally. For extensions, this kind of export behavior is not allowed at all.

---

<div class="post-metadata">

**Author:** ![hhaensel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hhaensel/32/1207_2.png) [@hhaensel](https://discourse.julialang.org/u/hhaensel)\
**Post date:** [April 20, 2023, 8:40am UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/14 "2023-04-20T08:40:36Z")

</div>

I’d like to learn more about this philosophy.  
I think, it makes totally sense to have certain types available only if two (or more) packages are loaded.  
I agree that they shouldn’t become part of the the parent module as it is the case with Require.jl, but leaving them in the extension module and allowing import/export should not be critical.

I could imagine that the difficulty is name space collision.

If the philosohpy is that the user shouldn’t know about the extension module or its name it’s not possible to import the types.  
People that tend to only import modules in order to keep their name space clean would need to know the name of the extension module.

So the best solution might be to provide a macro in the parent module that brings the newly defined types  
in the name space of the parent module.  
Users would need to explicitly call that macro.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [April 20, 2023, 12:48pm UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/15 "2023-04-20T12:48:02Z")

</div>

> [@hhaensel](#):
>
> In `Reqires.jl` it is possible to export variables. They are part of the parent module. For the extension module I have not been able to export or import variables, neither as part of the extension module, nor via `@eval <parentmodule> export x, y` as this would break precompilation.

You want to export symbols that are conditionally defined based on what packages have been loaded in the session? Just checking so I understand.

You could do something like:

```julia
module Parent

export get_thingy

function get_thingy()
    ext = Base.get_extension(@ __MODULE__ , :Extension)
    if ext === nothing
        error("extension not loaded...")
    else
        return ext.get_thingy()
    end
end

```

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [April 20, 2023, 1:02pm UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/16 "2023-04-20T13:02:57Z")

</div>

Can it be

```julia
module Parent
export get_thingy
get_thingy() = error("Load package XXX")
end

module SomeExt
using XXX
import Parent.get_thingy
get_thingy() = actual code
end

```

?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [April 20, 2023, 1:28pm UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/17 "2023-04-20T13:28:56Z")

</div>

The issue with that is that you are directly overwriting the method in Parent which is usually not a good idea.

---

<div class="post-metadata">

**Author:** ![hhaensel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hhaensel/32/1207_2.png) [@hhaensel](https://discourse.julialang.org/u/hhaensel)\
**Post date:** [April 20, 2023, 5:23pm UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/18 "2023-04-20T17:23:58Z")

</div>

Thanks for your answer, this is what I had in mind. But I would rather return a type than a function.

```julia
module Parent

export get_thingy, @get_thingy

function get_thingy()
    ext = Base.get_extension(@ __MODULE__ , :Extension)
    if ext === nothing
        error("extension not loaded...")
    else
        return ext.MyExtensionType
    end
end

macro get_thingy()
    ext = Base.get_extension(@ __MODULE__ , :Extension)
    MyType = if ext === nothing
        error("extension not loaded...")
    else
        ext.MyExtensionType
    end
    :(const MyExtensionType = $MyType) |> esc
end

# Main

const MyExtensionType = get_thingy()

# alternatively (not tested yet, but along these lines)
@get_thingy

```

I could imagine official extension macros `@import` and `@using` with the following syntax

```julia
# import var1 and var2 from ParentExtension1Extension2Ext
@import Parent, Extension1, Extension2: var1, var2 as myvar 2

# import all exports from ParentExtension1Extension2Ext
@using Parent, Extension1, Extension2

# import all exports from all loaded Extensions
@using Parent __all

```

Does something along these lines sound reasonable to you?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [April 21, 2023, 6:59am UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/19 "2023-04-21T06:59:40Z")

</div>

You cannot have `ext = Base.get_extension(@ __MODULE__ , :Extension)` outside the macro expansion, it needs to be inside the returned expression.

Extensions are a run time thing, they might be loaded or they might not be so you have to check. Therefore, you cannot make a decision about them during precompile time (or macro expansion time). That’s why you need to run some code (like a function) to interact with them and you cannot really just export a type from it.

---

<div class="post-metadata">

**Author:** ![hhaensel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hhaensel/32/1207_2.png) [@hhaensel](https://discourse.julialang.org/u/hhaensel)\
**Post date:** [April 26, 2023, 11:58pm UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/20 "2023-04-26T23:58:27Z")

</div>

Maybe I was not very clear with my last post. I did not mean to use the macro from within the extension.  
It could be called by the user after loading the parent and the extension module. This way the user could import variables or types from extensions without knowing the name of the extensions.

As an MWE I have prepared the following scenario:

- packages `Parent`, `Child1`, `Child2`
- extensions: ParentChild1Ext, ParentChild2Ext which export the types C1 and C2, respectively.

e.g.

```julia
module ParentChild1Ext

export C1
println("Child1 Extensions")

mutable struct C1
    a::Int
    b::Int
end

end

```

The following macro imports all exports of the extensions that have been loaded at that point of time

```julia
using TOML

macro import_extensions(parent, prefix = "")
    prefix isa QuoteNode && (prefix = String(prefix.value))
    toml = joinpath(dirname(dirname(pathof(@eval( __module__ , $parent)))), "Project.toml")
    extensions = get(TOML.parsefile(toml), "extensions", nothing)
    
    ee = [Base.get_extension(Parent, Symbol(k)) for k in keys(extensions)]
    ee = ee[ee .!== nothing]
    output = quote end
    for e in ee
        nn = setdiff(names(e), [Symbol(e)])
        for n in nn
            n_new = Symbol(prefix, isempty(prefix) ? "" : "_", n)
            if isdefined( __module__ , n_new) && @eval( __module__ , $n_new !== $e.$n)
                @warn "$(repr(n)) already exists and could not be imported"
            else
                push!(output.args, :(const $n_new = $e.$n))
            end
        end
    end
    push!(output.args, :nothing)
    :($output) |> esc
end

using Parent
using Child1

@import_extensions Parent
# imports C1

using Child2

@import_extensions Parent
# imports C2

@import_extensions Parent :hh
# imports C2 as hh_C2

```

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [April 27, 2023, 6:16am UTC](https://discourse.julialang.org/t/are-extension-packages-importable/92527/21 "2023-04-27T06:16:39Z")

</div>

Try

```julia
f() = @import_extensions Parent

using Parent
using Child1

f()

```

and it doesn’t work. You are relying on the state of Julia at the time the macro is expanded. You can of course do that but that makes it tricky to use in a package for example, when the extension will be loaded after the macro expansion.

[Next page](https://discourse.julialang.org/t/are-extension-packages-importable/92527.md?page=2)