# When does package code get loaded in a module

**URL:** <https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253>\
**Category:** General Usage\
**Created:** [November 16, 2020, 9:52pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253 "2020-11-16T21:52:01Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![louisponet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/louisponet/32/2070_2.png) [@louisponet](https://discourse.julialang.org/u/louisponet)\
**Post date:** [November 16, 2020, 9:52pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/1 "2020-11-16T21:52:01Z")

</div>

Hi all,  
Since my last question [world age problem when does a generated function get generated](https://discourse.julialang.org/t/world-age-problem-when-does-a-generated-function-get-generated/49302) did not get any answers, maybe because it was too specific, I’ll try again here.

I think my issue boils down to this question:

```julia
module TestMod
using Package1
using Package2
...

```

Can a generated function in `Package2` use a function that was defined in Package 1? If I try in a REPL it works since as the documentation on generated functions suggests, code that was defined in Package 1 before the generated function definition in Package 2 should be usable.

However, if I try it in the module itself I get a world age error. Is this the expected behavior? If so, should I understand it in that all code from `using PackageX` statements in a module is loaded at the same time (same world age)?

EDIT:  
MWE (This will create a subdirectory TestMod)

```julia
using Pkg
Pkg.generate("TestMod")
rm("TestMod/src/TestMod.jl")
open("TestMod/src/TestMod.jl", "w") do f
    write(f, """
module TestMod
using ColorTypes
using GLAbstraction
function test()
    GLAbstraction.textureformat_from_type(ColorTypes.RGBA{Float32})
end
end
""")
end
Pkg.activate("TestMod")
Pkg.add(url="https://github.com/JuliaGL/GLAbstraction.jl",rev="master")
Pkg.add("ColorTypes")
using TestMod
@show TestMod.test()

```

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [November 16, 2020, 10:03pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/2 "2020-11-16T22:03:41Z")

</div>

Can you be more specific about what your `Package1` and `Package2` look like? My naive attempt to have a generated function in Package2 use a function from Package1 works just fine:

```julia
# Package1.jl
module Package1

function f(x)
	x + 1
end

end

# Package2.jl
module Package2

using Package1

@generated function g(x)
	quote
		Package1.f(x) + 1
	end
end

end

# TestMod.jl
module TestMod

using Package1
using Package2

@assert Package2.g(1) == 3

end

```

Usage:

```julia
julia> push!(LOAD_PATH, @ __DIR__ )
4-element Array{String,1}:
 "@"
 "@v#.#"
 "@stdlib"
 "/home/user/Downloads/test"

julia> using TestMod
[Info: Precompiling TestMod [top-level]

```

I also tried using the function from `Package1` inside the _generator_ rather than the returned expression, and this also worked exactly as expected:

```julia
# Package2.jl
module Package2

using Package1

@generated function g(x)
	y = Package1.f(0)
	quote
		x + 1 + $y
	end
end

end

```

If you can be more specific about the actual error you’re seeing, then perhaps we can figure out what’s going on in your case.

---

<div class="post-metadata">

**Author:** ![louisponet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/louisponet/32/2070_2.png) [@louisponet](https://discourse.julialang.org/u/louisponet)\
**Post date:** [November 16, 2020, 10:07pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/3 "2020-11-16T22:07:02Z")

</div>

Okay, sorry I was supposing that Package 2 did not use Package 1.

In my case Package 2 is using a `length` function defined in Package 1 for some of it’s types.  
Specifically Package 1 is ColorTypes, defining `length(::Type{<:Colorant})`, where Package 2 is GLAbstraction using that length to generate the code for creating a Texture with those colors.  
I did not want GLAbstraction to depend on ColorTypes since I want the mechanism to be relatively general.

EDIT:  
[GLAbstraction line](https://github.com/JuliaGL/GLAbstraction.jl/blob/c8e6def22a85d23bf8c7a991e6d30f4a629aaae6/src/texture.jl#L58)

[ColorTypes line](https://github.com/JuliaGraphics/ColorTypes.jl/blob/bd31741d162361ebd44ed05ae532266998d9ce9f/src/types.jl#L428)

[Glimpse.jl using both packages](https://github.com/louisponet/Glimpse.jl/blob/master/src/Glimpse.jl)

This used to work before v1.5.x

Error message:

```julia
julia> Glimpse.GLAbstraction.textureformat_from_type(Glimpse.RGBA{Float32})
ERROR: MethodError: no method matching length(::Type{RGBA{Float32}})
The applicable method may be too new: running in world age 27846, while current world is 27903.
Closest candidates are:
  length(::Type{var"#s38"} where var"#s38"<:RGBA) at /home/ponet/.julia/environments/GlimpseDev/src/extensions.jl:304 (method too new to be called from this world context.)
  length(::Type{C}) where {N, C<:(Colorant{T,N} where T)} at /home/ponet/.julia/packages/ColorTypes/RF8lb/src/types.jl:429 (method too new to be called from this world context.)
  length(::Type{Glimpse.UniformColor}) at /home/ponet/.julia/environments/GlimpseDev/src/components.jl:121 (method too new to be called from this world context.)
  ...
Stacktrace:
 [1] cardinality(::Type{T} where T) at /home/ponet/.julia/dev/GLAbstraction/src/utils.jl:46
 [2] textureformat_from_type_sym(::Type{RGBA{Float32}}) at /home/ponet/.julia/dev/GLAbstraction/src/texture.jl:102
 [3] #s121#9 at /home/ponet/.julia/dev/GLAbstraction/src/texture.jl:107 [inlined]
 [4] #s121#9(::Any, ::Any, ::Any) at ./none:0
 [5] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:527
 [6] top-level scope at REPL[6]:1

```

---

<div class="post-metadata">

**Author:** ![louisponet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/louisponet/32/2070_2.png) [@louisponet](https://discourse.julialang.org/u/louisponet)\
**Post date:** [November 16, 2020, 10:31pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/4 "2020-11-16T22:31:46Z")

</div>

MWE is this:

```julia
# TestMod.jl
module TestMod
    using Package1
    using Package2
    function test()
        Package2.test(Package1.TestType)
    end
end

# Package1.jl
module Package1
    struct TestType end
    Base.length(::Type{TestType}) = 4
end

# Package2.jl
module Package2
@generated test(::Type{T}) where {T} = Symbol(length(T))
end

using TestMod
TestMod.test()

```

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [November 16, 2020, 10:37pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/5 "2020-11-16T22:37:43Z")

</div>

> Okay, sorry I was supposing that Package 2 did not use Package 1.

But then how would Package2 call a method from Package1 without actually using Package1? There’s no mechanism to do that, regardless of the use of `@generated` functions.

> [@louisponet](#):
>
> `length(::Type{var"#s38"} where var"#s38"<:RGBA) at /home/ponet/.julia/environments/GlimpseDev/src/extensions.jl:304 (method too new to be called from this world context.)`

This line is suspicious–it’s a method of `Base.length` on a type from `ColorTypes.jl` defined in some third location. That’s a classic example of “type piracy”, and this kind of confusion is exactly why type piracy is not recommended. It looks like `ColorTypes` already defines this method, so why do you need to repeat it elsewhere?

---

<div class="post-metadata">

**Author:** ![louisponet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/louisponet/32/2070_2.png) [@louisponet](https://discourse.julialang.org/u/louisponet)\
**Post date:** [November 16, 2020, 10:39pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/6 "2020-11-16T22:39:05Z")

</div>

You are right, this is a leftover of me trying to fix it by explicitly defining a length method for RGBA{Float32} in Glimpse.

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [November 16, 2020, 10:40pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/7 "2020-11-16T22:40:06Z")

</div>

> [@louisponet](#):
>
> MWE is this:

The error produced by that is:

```julia
julia> using TestMod
[Info: Precompiling TestMod [top-level]
ERROR: LoadError: UndefVarError: 4 not defined

```

but that’s not a world-age issue, it’s just that the `@generated` function `g` returns `Symbol(4)` as its quoted code. When the generated code is run, it tries to return the value of the variable named `:4`, which doesn’t exist.

If we change `g`, then everything works as intended:

```julia
module Package2
@generated test(::Type{T}) where {T} = :($(length(T)))
end

```

---

<div class="post-metadata">

**Author:** ![louisponet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/louisponet/32/2070_2.png) [@louisponet](https://discourse.julialang.org/u/louisponet)\
**Post date:** [November 16, 2020, 10:41pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/8 "2020-11-16T22:41:20Z")

</div>

Wow really? I get this error on v1.5.1:

```julia
julia> TestMod.test()
ERROR: MethodError: no method matching length(::Type{Package1.TestType})
The applicable method may be too new: running in world age 27818, while current world is 27820.
Closest candidates are:
  length(::Type{Package1.TestType}) at /home/ponet/.julia/environments/GlimpseDev/Package1/src/Package1.jl:3 (method too new to be called from this world context.)
  length(::BitSet) at bitset.jl:365
  length(::Base.MethodList) at reflection.jl:869
  ...
Stacktrace:
 [1] #s12#1 at /home/ponet/.julia/environments/GlimpseDev/Package2/src/Package2.jl:2 [inlined]
 [2] #s12#1(::Any, ::Any, ::Any) at ./none:0
 [3] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:527
 [4] test() at /home/ponet/.julia/environments/GlimpseDev/TestMod/src/TestMod.jl:5
 [5] top-level scope at REPL[4]:1

```

---

<div class="post-metadata">

**Author:** ![louisponet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/louisponet/32/2070_2.png) [@louisponet](https://discourse.julialang.org/u/louisponet)\
**Post date:** [November 16, 2020, 10:43pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/9 "2020-11-16T22:43:07Z")

</div>

I have generated real package directories for each of the modules though, is that a difference?

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [November 16, 2020, 10:43pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/10 "2020-11-16T22:43:29Z")

</div>

Well that’s weird. I get no error with Julia 1.5.1 and 1.5.3. Can you boil your MWE down into a single file? Having to maintain multiple different files makes it tricky to ensure we’re doing exactly the same thing in the same way.

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [November 16, 2020, 10:43pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/11 "2020-11-16T22:43:51Z")

</div>

> [@louisponet](#):
>
> I have generated real package directories for each of the modules though, is that a difference?

Possibly, but I can’t think of why it would matter. Very interesting indeed.

---

<div class="post-metadata">

**Author:** ![louisponet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/louisponet/32/2070_2.png) [@louisponet](https://discourse.julialang.org/u/louisponet)\
**Post date:** [November 16, 2020, 10:46pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/12 "2020-11-16T22:46:04Z")

</div>

Okay to reproduce it, you would do the following commands:  
In topdir:  
`pkg> generate Package1`  
`pkg> generate Package2`  
`pkg> generate TestMod`  
paste contents of the Package1.jl Package2.jl TestMod.jl into the generated module definitions inside each of the subdirs/src/  
`pkg> activate TestMod`  
`pkg> dev Package1`  
`pkg> dev Package2`  
`using TestMod; TestMod.test()`

---

<div class="post-metadata">

**Author:** ![louisponet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/louisponet/32/2070_2.png) [@louisponet](https://discourse.julialang.org/u/louisponet)\
**Post date:** [November 16, 2020, 11:04pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/13 "2020-11-16T23:04:27Z")

</div>

What’s even weirder is that if I run this file:

```julia
using Pkg
Pkg.generate("Package1")
Pkg.generate("Package2")
Pkg.generate("TestMod")
rm("Package1/src/Package1.jl")
rm("Package2/src/Package2.jl")
rm("TestMod/src/TestMod.jl")
open("Package1/src/Package1.jl", "w") do f
    write(f, """
module Package1
struct TestType end
Base.length(::Type{TestType}) = 4
end
""")
end
open("Package2/src/Package2.jl", "w") do f
    write(f, """
module Package2
@generated test(::Type{T}) where T = :(Symbol(length(T))) #I know this is wrong but I don't know how to write dollar
end
""")
end
open("TestMod/src/TestMod.jl", "w") do f
    write(f, """
module TestMod
using Package1
using Package2
function test()
    Package2.test(Package1.TestType)
end
end
""")
end
Pkg.activate("TestMod")
Pkg.develop(path="Package1")
Pkg.develop(path="Package2")
using TestMod
@show TestMod.test()

```

It doesn’t error… REPL world vs Module world thingy?

---

<div class="post-metadata">

**Author:** ![louisponet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/louisponet/32/2070_2.png) [@louisponet](https://discourse.julialang.org/u/louisponet)\
**Post date:** [November 16, 2020, 11:35pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/14 "2020-11-16T23:35:59Z")

</div>

I’m at a loss, I can’t seem to reproduce it anymore neither with the repl way, nor with the file, but the original issue persists. I went through my repl history and repeating the exact same steps don’t lead to the same error… I have added a MWE with my actual issue to the first post, I don’t know how it differs in essence from the one just above.

---

<div class="post-metadata">

**Author:** ![louisponet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/louisponet/32/2070_2.png) [@louisponet](https://discourse.julialang.org/u/louisponet)\
**Post date:** [November 17, 2020, 10:14am UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/15 "2020-11-17T10:14:28Z")

</div>

Okay wow, it stops working when using the desired `:($(length(T)))` instead of `:(Symbol(length(T)))`  
What is that all about.

```julia
using Pkg
ispath("Package1") && rm("Package1", recursive=true)
ispath("Package2") && rm("Package2", recursive=true)
ispath("TestMod") && rm("TestMod", recursive=true)
Pkg.generate("Package1")
Pkg.generate("Package2")
Pkg.generate("TestMod")
rm("Package1/src/Package1.jl")
rm("Package2/src/Package2.jl")
rm("TestMod/src/TestMod.jl")
open("Package1/src/Package1.jl", "w") do f
    write(f, """
module Package1
struct TestType end
Base.length(::Type{TestType}) = 4
end
""")
end
open("Package2/src/Package2.jl", "w") do f
    write(f, """
module Package2
@generated test(::Type{T}) where T = :(\$(length(T)))
end
""")
end
open("TestMod/src/TestMod.jl", "w") do f
    write(f, """
module TestMod
using Package1
using Package2
function test()
    Package2.test(Package1.TestType)
end
end
""")
end
Pkg.activate("TestMod")
Pkg.develop(path="Package1")
Pkg.develop(path="Package2")
using TestMod
@show TestMod.test()

```

Errors with world age

---

<div class="post-metadata">

**Author:** ![louisponet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/louisponet/32/2070_2.png) [@louisponet](https://discourse.julialang.org/u/louisponet)\
**Post date:** [November 17, 2020, 9:50pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/16 "2020-11-17T21:50:29Z")

</div>

For future reference with the help of @vchuravy I think I understood the issue (please correct me if I’m wrong):  
Packages that are being used inside a module get precompiled in arbitrary order, with each of them acquiring a world age after finishing the precompilation step. The order of `using PackageX` inside a module does _not_ matter at all.

This means that any extensions to, e.g. `Base.length` defined inside the used packages should be considered as nonexistent for the purpose of `@generated` functions inside other used packages.  
Using them might sometimes work, when world ages happen to align correctly, sometimes not, i.e. the behavior is undefined and should not be relied upon.

The scary and bad way around this is by using `Core._apply_pure`, the better way is to communicate constants to the generated functions by using `Val`.  
As a (trivial) example:

```julia
@generated function foo(::Type{T}) where {T}
    if Core._apply_pure(length, (T,)) == 4
       return :(4+4)
   else 
      return :(1+1)
end

```

can _and should_ be replaced by

```julia
foo(::Type{T}) = foo(Val{length(T)}())

@generated function foo(::Val{length}) where {length}
    if length == 4
       return :(4+4)
   else 
      return :(1+1)
end

```

Afaik there is (usually) no (practical) difference between the first and second method, both get lowered to a const evaluation.

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [November 17, 2020, 10:17pm UTC](https://discourse.julialang.org/t/when-does-package-code-get-loaded-in-a-module/50253/17 "2020-11-17T22:17:54Z")

</div>

Ah, yeah, that makes sense. I guess the original issue could also be described as the generator relying on non-const global state (the global method table).

Thanks for writing this up. I’m sure this will be a useful post to refer to later.