When does package code get loaded in a module

General Usage
1

Hi all,
Since my last question world age problem when does a generated function get generated did not get any answers, maybe because it was too specific, I’ll try again here.

I think my issue boils down to this question:

module TestMod
using Package1
using Package2
...

Can a generated function in Package2 use a function that was defined in Package 1? If I try in a REPL it works since as the documentation on generated functions suggests, code that was defined in Package 1 before the generated function definition in Package 2 should be usable.

However, if I try it in the module itself I get a world age error. Is this the expected behavior? If so, should I understand it in that all code from using PackageX statements in a module is loaded at the same time (same world age)?

EDIT:
MWE (This will create a subdirectory TestMod)

using Pkg
Pkg.generate("TestMod")
rm("TestMod/src/TestMod.jl")
open("TestMod/src/TestMod.jl", "w") do f
    write(f, """
module TestMod
using ColorTypes
using GLAbstraction
function test()
    GLAbstraction.textureformat_from_type(ColorTypes.RGBA{Float32})
end
end
""")
end
Pkg.activate("TestMod")
Pkg.add(url="https://github.com/JuliaGL/GLAbstraction.jl",rev="master")
Pkg.add("ColorTypes")
using TestMod
@show TestMod.test()
2

Can you be more specific about what your Package1 and Package2 look like? My naive attempt to have a generated function in Package2 use a function from Package1 works just fine:

# Package1.jl
module Package1

function f(x)
	x + 1
end

end

# Package2.jl
module Package2

using Package1

@generated function g(x)
	quote
		Package1.f(x) + 1
	end
end

end

# TestMod.jl
module TestMod

using Package1
using Package2

@assert Package2.g(1) == 3

end

Usage:

julia> push!(LOAD_PATH, @__DIR__)
4-element Array{String,1}:
 "@"
 "@v#.#"
 "@stdlib"
 "/home/user/Downloads/test"

julia> using TestMod
[ Info: Precompiling TestMod [top-level]

I also tried using the function from Package1 inside the generator rather than the returned expression, and this also worked exactly as expected:

# Package2.jl
module Package2

using Package1

@generated function g(x)
	y = Package1.f(0)
	quote
		x + 1 + $y
	end
end

end

If you can be more specific about the actual error you’re seeing, then perhaps we can figure out what’s going on in your case.

1 Like
3

Okay, sorry I was supposing that Package 2 did not use Package 1.

In my case Package 2 is using a length function defined in Package 1 for some of it’s types.
Specifically Package 1 is ColorTypes, defining length(::Type{<:Colorant}), where Package 2 is GLAbstraction using that length to generate the code for creating a Texture with those colors.
I did not want GLAbstraction to depend on ColorTypes since I want the mechanism to be relatively general.

EDIT:
GLAbstraction line

ColorTypes line

Glimpse.jl using both packages

This used to work before v1.5.x

Error message:

julia> Glimpse.GLAbstraction.textureformat_from_type(Glimpse.RGBA{Float32})
ERROR: MethodError: no method matching length(::Type{RGBA{Float32}})
The applicable method may be too new: running in world age 27846, while current world is 27903.
Closest candidates are:
  length(::Type{var"#s38"} where var"#s38"<:RGBA) at /home/ponet/.julia/environments/GlimpseDev/src/extensions.jl:304 (method too new to be called from this world context.)
  length(::Type{C}) where {N, C<:(Colorant{T,N} where T)} at /home/ponet/.julia/packages/ColorTypes/RF8lb/src/types.jl:429 (method too new to be called from this world context.)
  length(::Type{Glimpse.UniformColor}) at /home/ponet/.julia/environments/GlimpseDev/src/components.jl:121 (method too new to be called from this world context.)
  ...
Stacktrace:
 [1] cardinality(::Type{T} where T) at /home/ponet/.julia/dev/GLAbstraction/src/utils.jl:46
 [2] textureformat_from_type_sym(::Type{RGBA{Float32}}) at /home/ponet/.julia/dev/GLAbstraction/src/texture.jl:102
 [3] #s121#9 at /home/ponet/.julia/dev/GLAbstraction/src/texture.jl:107 [inlined]
 [4] #s121#9(::Any, ::Any, ::Any) at ./none:0
 [5] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:527
 [6] top-level scope at REPL[6]:1
4

MWE is this:

# TestMod.jl
module TestMod
    using Package1
    using Package2
    function test()
        Package2.test(Package1.TestType)
    end
end

# Package1.jl
module Package1
    struct TestType end
    Base.length(::Type{TestType}) = 4
end

# Package2.jl
module Package2
@generated test(::Type{T}) where {T} = Symbol(length(T))
end

using TestMod
TestMod.test()
5

Okay, sorry I was supposing that Package 2 did not use Package 1.

But then how would Package2 call a method from Package1 without actually using Package1? There’s no mechanism to do that, regardless of the use of @generated functions.

This line is suspicious–it’s a method of Base.length on a type from ColorTypes.jl defined in some third location. That’s a classic example of “type piracy”, and this kind of confusion is exactly why type piracy is not recommended. It looks like ColorTypes already defines this method, so why do you need to repeat it elsewhere?

1 Like
6

You are right, this is a leftover of me trying to fix it by explicitly defining a length method for RGBA{Float32} in Glimpse.

7

The error produced by that is:

julia> using TestMod
[ Info: Precompiling TestMod [top-level]
ERROR: LoadError: UndefVarError: 4 not defined

but that’s not a world-age issue, it’s just that the @generated function g returns Symbol(4) as its quoted code. When the generated code is run, it tries to return the value of the variable named :4, which doesn’t exist.

If we change g, then everything works as intended:

module Package2
@generated test(::Type{T}) where {T} = :($(length(T)))
end
1 Like
8

Wow really? I get this error on v1.5.1:

julia> TestMod.test()
ERROR: MethodError: no method matching length(::Type{Package1.TestType})
The applicable method may be too new: running in world age 27818, while current world is 27820.
Closest candidates are:
  length(::Type{Package1.TestType}) at /home/ponet/.julia/environments/GlimpseDev/Package1/src/Package1.jl:3 (method too new to be called from this world context.)
  length(::BitSet) at bitset.jl:365
  length(::Base.MethodList) at reflection.jl:869
  ...
Stacktrace:
 [1] #s12#1 at /home/ponet/.julia/environments/GlimpseDev/Package2/src/Package2.jl:2 [inlined]
 [2] #s12#1(::Any, ::Any, ::Any) at ./none:0
 [3] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:527
 [4] test() at /home/ponet/.julia/environments/GlimpseDev/TestMod/src/TestMod.jl:5
 [5] top-level scope at REPL[4]:1
9

I have generated real package directories for each of the modules though, is that a difference?

10

Well that’s weird. I get no error with Julia 1.5.1 and 1.5.3. Can you boil your MWE down into a single file? Having to maintain multiple different files makes it tricky to ensure we’re doing exactly the same thing in the same way.

11

Possibly, but I can’t think of why it would matter. Very interesting indeed.

12

Okay to reproduce it, you would do the following commands:
In topdir:
pkg> generate Package1
pkg> generate Package2
pkg> generate TestMod
paste contents of the Package1.jl Package2.jl TestMod.jl into the generated module definitions inside each of the subdirs/src/
pkg> activate TestMod
pkg> dev Package1
pkg> dev Package2
using TestMod; TestMod.test()

13

What’s even weirder is that if I run this file:

using Pkg
Pkg.generate("Package1")
Pkg.generate("Package2")
Pkg.generate("TestMod")
rm("Package1/src/Package1.jl")
rm("Package2/src/Package2.jl")
rm("TestMod/src/TestMod.jl")
open("Package1/src/Package1.jl", "w") do f
    write(f, """
module Package1
struct TestType end
Base.length(::Type{TestType}) = 4
end
""")
end
open("Package2/src/Package2.jl", "w") do f
    write(f, """
module Package2
@generated test(::Type{T}) where T = :(Symbol(length(T))) #I know this is wrong but I don't know how to write dollar
end
""")
end
open("TestMod/src/TestMod.jl", "w") do f
    write(f, """
module TestMod
using Package1
using Package2
function test()
    Package2.test(Package1.TestType)
end
end
""")
end
Pkg.activate("TestMod")
Pkg.develop(path="Package1")
Pkg.develop(path="Package2")
using TestMod
@show TestMod.test()

It doesn’t error… REPL world vs Module world thingy?

14

I’m at a loss, I can’t seem to reproduce it anymore neither with the repl way, nor with the file, but the original issue persists. I went through my repl history and repeating the exact same steps don’t lead to the same error… I have added a MWE with my actual issue to the first post, I don’t know how it differs in essence from the one just above.

15

Okay wow, it stops working when using the desired :($(length(T))) instead of :(Symbol(length(T)))
What is that all about.

using Pkg
ispath("Package1") && rm("Package1", recursive=true)
ispath("Package2") && rm("Package2", recursive=true)
ispath("TestMod") && rm("TestMod", recursive=true)
Pkg.generate("Package1")
Pkg.generate("Package2")
Pkg.generate("TestMod")
rm("Package1/src/Package1.jl")
rm("Package2/src/Package2.jl")
rm("TestMod/src/TestMod.jl")
open("Package1/src/Package1.jl", "w") do f
    write(f, """
module Package1
struct TestType end
Base.length(::Type{TestType}) = 4
end
""")
end
open("Package2/src/Package2.jl", "w") do f
    write(f, """
module Package2
@generated test(::Type{T}) where T = :(\$(length(T)))
end
""")
end
open("TestMod/src/TestMod.jl", "w") do f
    write(f, """
module TestMod
using Package1
using Package2
function test()
    Package2.test(Package1.TestType)
end
end
""")
end
Pkg.activate("TestMod")
Pkg.develop(path="Package1")
Pkg.develop(path="Package2")
using TestMod
@show TestMod.test()

Errors with world age

16

For future reference with the help of @vchuravy I think I understood the issue (please correct me if I’m wrong):
Packages that are being used inside a module get precompiled in arbitrary order, with each of them acquiring a world age after finishing the precompilation step. The order of using PackageX inside a module does not matter at all.

This means that any extensions to, e.g. Base.length defined inside the used packages should be considered as nonexistent for the purpose of @generated functions inside other used packages.
Using them might sometimes work, when world ages happen to align correctly, sometimes not, i.e. the behavior is undefined and should not be relied upon.

The scary and bad way around this is by using Core._apply_pure, the better way is to communicate constants to the generated functions by using Val.
As a (trivial) example:

@generated function foo(::Type{T}) where {T}
    if Core._apply_pure(length, (T,)) == 4
       return :(4+4)
   else 
      return :(1+1)
end

can and should be replaced by

foo(::Type{T}) = foo(Val{length(T)}())

@generated function foo(::Val{length}) where {length}
    if length == 4
       return :(4+4)
   else 
      return :(1+1)
end

Afaik there is (usually) no (practical) difference between the first and second method, both get lowered to a const evaluation.

2 Likes
17

Ah, yeah, that makes sense. I guess the original issue could also be described as the generator relying on non-const global state (the global method table).

Thanks for writing this up. I’m sure this will be a useful post to refer to later.

1 Like