How to properly implement lazy compilation of submodules? Problem with invokelatest for kwarg functions

General Usage
,
1

Inspired by LazyModules I’m trying to make delayed loading of some of submodules from my package. However, it implies that at some point I need to update a module definition on the fly. It works well with normal functions, but by some reason using keyword arguments breaks Base.invokelatest.
Below is a simplified example (still long, sorry, but nothing shorter reproduces the desired behavior).

module T

# Code for lazy submodules

mutable struct LazySubmodule
    _module_name::Symbol
    _code::Expr
    _module::Union{Module, Nothing}
end

macro lm(ex)
    alias, code = ex.args
    s = LazySubmodule(alias, code, __module__)
    Core.eval(__module__, :($(alias) = $s))
end

function Base.getproperty(m::LazySubmodule, s::Symbol)
    if s in (:_module_name, :_code, :_module)
        return getfield(m, s)
    end

    # Compile the sub-module
    Core.eval(m._module, :($(m._code)))
    alias = m._code.args[2]
    # Rewrite it's alias with the compiled module instead of the LazySubmodule
    mod = Core.eval(m._module, :($(m._module_name) = $(alias)))

    f = getfield(mod, s)
    # invokelatest should solve the issue with world age and missing functions
    return (args...; kwargs...) -> Base.invokelatest(f, args...; kwargs...)
end

# Define my submodules

@lm M1 = module Module1
    f1(s::String) = print(s)
end

f(s::String) = M1.f1(s)

end

This code doesn’t have any keyword arguments, and if I call T.f("Hello"), it prints “Hello” as expected. However, if I add keyword arguments to the module definitions as below, the code breaks.

@lm M1 = module Module1
    f1(;s::String) = print(s)
end

f(s::String) = M1.f1(s=s)

Now, the first call T.f("Hello") fails with error

ERROR: MethodError: no method matching f1(; s="Hello")
Closest candidates are:
  f1(; s)

while the second call works fine. It looks exactly like world age problems, and if I remove invokelatest, the same behavior could be observed for the case 1 (no keyword arguments), as well.

Could anyone please explain why this is different with and without kwargs, and how it could be fixed? I can see that the implementation of invokelatest indeed processes functions with kwargs differently. But the actual differences are in the core, which I’m unable to comprehend…

If there is a better way to make lazy loading of submodules, that would also solve the problem. :slight_smile:

UPDATE
The problem turns out to be not in invokelatest, but somewhere in how julia calls methods. Setting

function Base.getproperty(m::LazySubmodule, s::Symbol)
    if s in (:_module_name, :_code, :_module)
        return getfield(m, s)
    end

    Core.eval(m._module, :($(m._code)))
    alias = m._code.args[2]
    mod = Core.eval(m._module, :($(m._module_name) = $(alias)))

    return (args...; kwargs...) -> error("This code is never reached")
end

shows that the result of the function is never used, and instead it’s substituted with T.M1.f1 directly.

UPDATE 2
The problem is in how Julia insides convert keyword functions to positional functions. While the lowered code for positional arguments looks fine:

@code_lowered T.f("A")
CodeInfo(
1 ─ %1 = Base.getproperty(Main.T.M1, :f1)
│   %2 = (%1)(s)
└──      return %2
)

the code for functions with keywords is more complicated:

CodeInfo(
1 ─ %1 = (:s,)
│   %2 = Core.apply_type(Core.NamedTuple, %1)
│   %3 = Core.tuple(s)
│   %4 = (%2)(%3)
│   %5 = Base.getproperty(Main.T.M1, :f1)
│   %6 = Core.kwfunc(%5)
│   %7 = Base.getproperty(Main.T.M1, :f1)
│   %8 = (%6)(%4, %7)
└──      return %8
)

So, essentially it calls Base.getproperty twice. And as the module gets replaced between the two calls, only the first call uses invokelatest. Still, I’d appreciate any advise on how to deal with that.