# We Do Need Exportable New Structs and Functions in Extension

**URL:** <https://discourse.julialang.org/t/we-do-need-exportable-new-structs-and-functions-in-extension/105810>\
**Category:** General Usage\
**Tags:** question, package, package-extensions\
**Created:** [November 5, 2023, 6:20am UTC](https://discourse.julialang.org/t/we-do-need-exportable-new-structs-and-functions-in-extension/105810 "2023-11-05T06:20:42Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sole](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sole/32/202974_2.png) [@Sole](https://discourse.julialang.org/u/Sole)\
**Post date:** [November 5, 2023, 6:20am UTC](https://discourse.julialang.org/t/we-do-need-exportable-new-structs-and-functions-in-extension/105810/1 "2023-11-05T06:20:42Z")

</div>

I’m not the last man struggling with transfering from `Requires` pkg to officially supported extension feature for `Julia1.9+`.

With `Requires` pkg we don’t care about where a `function` and a `struct` are defined because we can easily access them. But when switching to extension feature, we can not access those **only** defined inside an extension `module`, to overcome this, one may need to pre-define an empty function or struct in the main pkg.

Let’s think about a using case. Suppose we have a `Person` module, and this `Person` has a lot of skills like `Driving`, `Writing` and `Boxing` etc. If `Driving` is really heavy, `Person` only needs a small part of it, let’s say driving a car, then we can make `Driving` as an `[weakdeps]`. To use `speedup` function **only** really implemented inside `PersonDrivingExt`, we need to declare an empty `speedup` in main pkg `Person` like

```julia
module Person

export speedup
function speedup end

export DrivingCarStatus
mutable struct DrivingCarStatus
    pos :: Coords
    speed :: Real
    function DrivingCarStatus(args...)
        # some construction here like
        new(initpos(xargs...), initspeed(yargs...))
    end
end

end # module Person

```

and then import and implement `speedup` in extension

```julia
module PersonDrivingExt

using Person, Driving
import Person: speedup, DrivingCarStatus

function speedup(status::DrivingCarStatus)
    status.speed += 10.0
end

end # module PersonDrivingExt

```

So far it is acceptable doing those. Well some issues may occur :

1. 😂 what if some types are only defined in `Driving`, like `::Coords` inside `DrivingCarStatus`
2. 😂 what if some functions are only defined in `Driving`, like `initpos` and `initspeed` in constructor

when these issues happen, we can `using Driving` inside `Person` to solve, but it 's far away from extension’s original purpose. _`"Extension is mainly for extending functions already defined in main pkg"`_ is apparently not enough and not convinient.

On my opinion, extension is good for compatibility control for weakdeps, but if `Requires` pkg can itself control the version of weakdeps, we could eassily merge some functionalities from heavy pkg dependency without changing much code while not introducing the extension feature from Julia1.9+. Treating a conditionaly-triggered-extension like an nature module would be nice, e.g.

```julia
module Person

# some Person's own functions and types here

end # module Person

# ======================================================

module PersonDrivingExt

  using Person, Driving
  export speedup, DrivingCarStatus

  mutable struct DrivingCarStatus
      pos :: Coords
      speed :: Real
      function DrivingCarStatus(args...)
          # some construction here like
          new(initpos(xargs...), initspeed(yargs...))
      end
  end

  function speedup(status::DrivingCarStatus)
      status.speed += 10.0
  end

end # module PersonDrivingExt

```

What new opinions do you have ? 😃

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [November 5, 2023, 6:56am UTC](https://discourse.julialang.org/t/we-do-need-exportable-new-structs-and-functions-in-extension/105810/2 "2023-11-05T06:56:02Z")

</div>

Full disclosure, I haven’t needed to write a package extension, yet. But first impressions, importing names originating in an extension sounds like a nightmare. I wouldn’t want to do `using Draw, Rainbow: DrawRainbowType` to get a name `DrawRainbowType` that belongs to neither `Draw` nor `Rainbow` alone. Extensions can’t be imported, just implicitly loaded, so if you somehow work around that to export new names, there’s no good way to organize them. There’s no reasonable choice of one package over the others when categorizing names into module instances (`parentmodule(DrawRainbowType), names(Draw)`) or documentations. If names were associated with and documented for the extension separately, I would rather that be an explicitly importable package `DrawRainbowPlus` that depends on `Draw` and `Rainbow`; this is much more accessible and a lot less misleading. After all, modules are Julia’s namespaces, so it’s intuitive that names that don’t originate in 2 spaces belong to a 3rd space that is handled the same. On the other hand, mixing names originating in `Draw` and `Rainbow` is more appropriate in an implicit extension than in a 3rd package dedicated to type piracy and re-exports. Making the unreasonable choice of one of the packages to export new names is not good either; it goes against modular programming principles for a module to expose conditionally existing names, especially if they depend on other modules’ independent existences.

There is a [documentation issue](https://github.com/JuliaLang/Pkg.jl/pull/3552) describing how to access extension-internal names, and you optionally bind to new variables instead of importing names. It looks awkward enough that it doesn’t seem intended for general usage. If it helps, here’s an example of how new variables differ from imported names, [without involving extensions](https://discourse.julialang.org/t/rename-just-one-of-imported-names-with-using/105026/13).

---

<div class="post-metadata">

**Author:** ![Sole](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sole/32/202974_2.png) [@Sole](https://discourse.julialang.org/u/Sole)\
**Post date:** [November 6, 2023, 8:35am UTC](https://discourse.julialang.org/t/we-do-need-exportable-new-structs-and-functions-in-extension/105810/3 "2023-11-06T08:35:04Z")

</div>

Thanks for your reply. 😃

As to this statment:

> Extensions can’t be imported, just implicitly loaded

I’d like to wonder _“is it designed like this ?”_ or _“is it techenically can’t be implemented ?”_ .

Age before `Requires.jl` pkg, pkg developers usually avoided loading a **heavy** dependency by creating another pkg, e.g. `CuYao.jl`, so when users had no nvidia-cuda device supporting, they could just use a cpu version `Yao.jl`. At the namespace of `Main` Module, `Yao` and `CuYao` are at the same hierarchy, we couldn’t say `Yao` is parent module and `CuYao` is child module. Bad thing to an end-user is that one has to read two pkgs’ documents when just using `CuYao`. Worse things would happen if we want MetalYao, AMDGPUYao, TPUYao, PlotYao, and WebYao etc.

Still take `CuYao` for example, there are two kinds of relationships between `CUDA`, `Yao` and `CuYao` at my point of view :

- 👁️‍🗨️ All three modules are of the same level just like three brothers. So creating a new function or a new type inside `CuYao` is easily understood as: **`CuYao` learned some good skills or belief from both `CUDA` and `Yao`, then formed his own ones.**

- 👁️‍🗨️`CuYao` is a child of `CUDA` and `Yao`. So creating a new function or a new type inside `CuYao` is easily understood as: **some gene (type) of `CUDA` + some gene (type) of `Yao` → new gene (type) of CuYao**.

Combing two pkg’s features to create new features is a common thing, we can still make it working by creating some empty API functions in `Yao` and then implement them in `CuYao`, but it’s difficulty to new structs that have types from `CUDA` or construtor having functions from `CUDA` for Julia1.9+.

As a end-user, I hope `CUDA` is just a trigger of `CuYao`, the version of `CUDA` is controlled by `Yao`, and to implicitlly load `CuYao` we put it in `ext` dir of `Yao`. In this way, user reads the combined hand-wirtten-markdown-docs in `Yao` without noticing the existence of `CuYao`.

PS: I’m not a user of `Yao` pkg, just use the three pkg names for convinience.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [November 6, 2023, 9:15am UTC](https://discourse.julialang.org/t/we-do-need-exportable-new-structs-and-functions-in-extension/105810/4 "2023-11-06T09:15:51Z")

</div>

> [@Sole](#):
>
> I’d like to wonder _“is it designed like this ?”_ or _“is it techenically can’t be implemented ?”_ .

My guess is that it is by design because Requires.jl already almost implements it; despite not being its intention, you can reexport names originating from a `@require` expression, and the expression can import a precompiled glue package. It thus seems very possible that extensions could have exported names, but the opposite direction was taken by making extensions not importable. I believe the reason is to avoid exposing conditionally existing names, which start to defeat the purpose of modules.

Again, I think that CuYao should be an explicitly imported package that owns the new names and depends on CUDA and Yao, and [it is](https://github.com/QuantumBFS/CuYao.jl/blob/master/Project.toml). Yes, it is shorter to write `using CUDA, Yao` than it is to write `using CUDA, Yao, CuYao` (well, CuYao already [reexports Yao](https://github.com/QuantumBFS/CuYao.jl/blob/master/src/CuYao.jl) so Yao is redundant), but its apparent convenience actually invites confusion. Say my package implicitly loads CuYao with `using CUDA, Yao` or `import CUDA, Yao: curand_state`; this is already terrible because specifying names in a multiple import was disallowed due to ambiguity e.g. `import Dates, BenchmarkTools: @btime`. A user later checks `parentmodule(curand_state)`, and one of two things can happen: 1) They get an implicitly imported `CuYao`, but they can’t find `import CuYao` or `module CuYao` in my source code or see a CuYao dependency in my Project.toml until they stumble down a long list into CUDA and Yao’s Project.toml files, 2) CuYao is not an importable name at all, in which case they might get an unhelpful `nothing`. By saving writing `CuYao`, it became much harder to follow where names come from.

---

<div class="post-metadata">

**Author:** ![Sole](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sole/32/202974_2.png) [@Sole](https://discourse.julialang.org/u/Sole)\
**Post date:** [November 6, 2023, 2:28pm UTC](https://discourse.julialang.org/t/we-do-need-exportable-new-structs-and-functions-in-extension/105810/5 "2023-11-06T14:28:30Z")

</div>

> [@Benny](#):
>
> By saving writing `CuYao`, it became much harder to follow where names come from.

Hey Benny, I think end-users would not care much about where the **name** comes from in the source code, instead they more need a well documented API about this name (new type or fn), and this can be wirtten into the official docs just like PyTorch docs.

I think make/pretend `Yao` as if it’s a whole fully stand alone pkg controlling most things it’s clear/clean to end-users, even `Yao` may have lots of helping pkgs hidden behind, like MetalYao, AMDGPUYao, TPUYao, PlotYao, and WebYao etc. If those helping pkgs are stand alone pkgs, users may feel messy.

Well, I may use two solutions to solve this:

1. 👾 Not using extentions provide by Julia1.9+, instead we keep using stand alone pkg as aid e.g. `CuYao`. `Yao`’s official document would say _“If you want a cuda accelaration, `using CuYao`, and if you want to plot something, `using PlotYao`, blabla…”_, then `CuYao` and `PlotYao` don’t even need official documents. In this way, end-users need to do `using Yao, CuYao`

2. 👾 Using a fake extension `CuYaoEmptyExt` to control CUDA’s version with Julia1.9+, and using `Requires.jl` to implicitily load `CuYao`, then `CuYao` can make new types and even export them. In this way, end-users only need to do `using Yao, CUDA`. Tricky, isn’t it ? 🤣 😂 😅 ☺

---

<div class="post-metadata">

**Author:** ![kellertuer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kellertuer/32/220707_2.png) [@kellertuer](https://discourse.julialang.org/u/kellertuer)\
**Post date:** [November 6, 2023, 3:27pm UTC](https://discourse.julialang.org/t/we-do-need-exportable-new-structs-and-functions-in-extension/105810/6 "2023-11-06T15:27:39Z")

</div>

I have written quite a few extensions by now and would maybe summarise this as

If you enter the regime where you need new structs – the scheme of a new package “between” the two you want to extend, might be a better choice. Basically as soon as something defines its own structs it should be its own package – I think Benny stated the reason: Conditionally existing names might be confusing.

I mainly used it to combine functions from one package (say `A.jl`) to work on structs from the other (say `B.jl`) as an extension to `A`. So if I would have structs from A as well that extend functions in B that would be something for an extension in B maybe?  
And for sure these new extensions (a) needed some rewrite of old Requires-based approaches and (b) are still using Requires as a fallback in pre 1.9. But after I got used to the thinking of extensions being for extending functionality (but not defining new functions or structs) – these work really great.

---

<div class="post-metadata">

**Author:** ![Sole](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sole/32/202974_2.png) [@Sole](https://discourse.julialang.org/u/Sole)\
**Post date:** [November 6, 2023, 4:04pm UTC](https://discourse.julialang.org/t/we-do-need-exportable-new-structs-and-functions-in-extension/105810/7 "2023-11-06T16:04:04Z")

</div>

> [@kellertuer](#):
>
> extensions being for extending functionality (but not defining new functions or structs)

Yeah, the extension feature of Julia1.9+ is probabaly 95% designed for this scenario. New structs is just a corner case out of expectation.

But if creating new struct is satisfied, it would be nice, then at most times we pkg-developers only extending existed functions, but we still give the corner case a chance to live. 😃

---

<div class="post-metadata">

**Author:** ![kellertuer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kellertuer/32/220707_2.png) [@kellertuer](https://discourse.julialang.org/u/kellertuer)\
**Post date:** [November 6, 2023, 5:00pm UTC](https://discourse.julialang.org/t/we-do-need-exportable-new-structs-and-functions-in-extension/105810/8 "2023-11-06T17:00:45Z")

</div>

I would carefully argue that if the code you mean to use as an extension has a new struct inside – it does not fit the scheme. Similar to the case where you try to do OOP in Julia – it can be done but it probably should not.

And sure there are corner cases, where a solution should carefully be discussed.  
For example, for my cache (where I have an extension using LRUCaches) I first had a struct working with the cache in the extension. Then I noticed I could do the same struct in the main package, when assuming that one field is “cache-like” and documenting what I expect from that field in functionality. With the extension that field of the struct is then just the LRUCache.

---

<div class="post-metadata">

**Author:** ![dubosipsl](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dubosipsl/32/52998_2.png) [@dubosipsl](https://discourse.julialang.org/u/dubosipsl)\
**Post date:** [November 6, 2023, 5:46pm UTC](https://discourse.julialang.org/t/we-do-need-exportable-new-structs-and-functions-in-extension/105810/9 "2023-11-06T17:46:17Z")

</div>

In your example, you may prefer to define `DrivingCarStatus` in module `Person` as an abstract type:

```julia
module Person

# implemented in PersonDrivingExt
export DrivingCarStatus, speedup
function speedup end
abstract type DrivingCarStatus end

end # module Person

```

Then in `PersonDrivingExt` you define `DrivingCarStatus` as a subtype of `Person.DrivingCarStatus` and define a constructor method:

```julia
module PersonDrivingExt
import Person
using Driving

mutable struct DrivingCarStatus <: Person.DrivingCarStatus
    pos :: Coords
    speed :: Real
    function DrivingCarStatus(args...)
        # some construction here like
        new(initpos(xargs...), initspeed(yargs...))
    end
end
Person.DrivingCarStatus(args...) = DrivingCarStatus(args...)

function Person.speedup(status::DrivingCarStatus)
    status.speed += 10.0
end

end # module PersonDrivingExt

```

From the user perspective, `using Person` is sufficient to dispatch on `DrivingCarStatus` and construct `DrivingCarStatus` instances.

You can put docstrings in `Person` so that they are available even when `PersonDrivingExt` is not loaded.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [November 6, 2023, 10:31pm UTC](https://discourse.julialang.org/t/we-do-need-exportable-new-structs-and-functions-in-extension/105810/10 "2023-11-06T22:31:02Z")

</div>

> [@Sole](#):
>
> Hey Benny, I think end-users would not care much about where the **name** comes from in the source code, and this can be wirtten into the official docs just like PyTorch docs.

As a end-user, I care. When I look up code snippets in forums, import statements are often all I have at the beginning to find the official docs. Names and behaviors shouldn’t depend on multiple loaded packages because it hampers reproducibility. Say I am given a code snippet `import Plots; Plots.plotyao(...)`, but it errors when I try it. Turns out the poster neglected to mention the “obvious” `using Yao` beforehand. Say I am given a code snippet `using Render3D; render(...)` but it runs incredibly slow. Turns out the poster neglected to copy and paste the `using CUDA` cell from their notebook.

> [@Sole](#):
>
> using `Requires.jl` to implicitily load `CuYao`, then `CuYao` can make new types and even export them. In this way, end-users only need to do `using Yao, CUDA`. Tricky, isn’t it ? 🤣 😂 😅 ☺

It is, actually. Who extended who to provide `curand_state` in `using Yao, CUDA, Plots`? I don’t think it’s reasonable to require that I read 3 docs sites or project files to figure that out, it should be apparent in the source code, and that’s only possible if each name is provided by 1 package unconditionally. How would I modify a provided snippet to import a specific name? `using Yao, CUDA, Plots: curand_state` throws an error. Even if you make it work, which package uses Requires.jl to load `curand_state`, or are they wholly independent packages that don’t use Requires.jl? I can’t tell from the statement at all. Note that Requires.jl unambiguously associates any conditional names with 1 of the packages; should `Yao.curand_state` be documented in Yao.jl but clarify “this doesn’t exist unless you also imported CUDA.jl”? How many such conditional names can bloat a package’s documentation, and how should they be organized if we generalize to extensions of \>2 packages? I’ll import a separate CuYao.jl, it’s so much easier to handle.

Packages are not only a way to load code but also to categorize names; dismissing that to save writing a few package names may be a miniscule convenience to users already familiar with the ecosystem, but it’ll definitely be a massive burden to users learning and sharing the libraries, which doesn’t seem like a good tradeoff.

> [@Sole](#):
>
> In this way, end-users need to do `using Yao, CuYao`

You don’t, like I’ve said, CuYao imports both CUDA and Yao and reexports Yao. `using CuYao` is much simpler and clearer than `using CUDA, Yao`, and specifying names actually works `using CuYao: curand_state`.