Dynamic Code Loading to Use Packages on Demand

General Usage
1

I ran into an interesting problem today that I am trying to figure out (along with @Wikunia and @arsh). Basically, it stems from the idea of trying to load specific packages by user demand. The specific issue we are working on is this: https://github.com/Wikunia/Javis.jl/pull/236

Here is what we are trying to achieve:

Package A has function main() which can call function extra() which resides in script extra_file.jl. The functions that are in extra_file.jl rely on package B. Package B however, is quite large and we desire to not use it each time a user runs A.main() as they may not always need main() to call function extra().

Some pseudo code could be thought of as follows (NOTE: ignore the fact that each time the user executes main(true) it uses B and includes extra_file.jl repeatedly. In practice, we only want to execute these commands once):

module A
    ...
    function main(call_extra = false)
        ...
        if call_extra == true
            using B
            include("extra_file.jl")
            extra()
        end
    end
    ...
end

We attempted to use Requires.jl to handle this problem but, to carry from the example, extra() was not able to be called. Then, we tried the exact same solution in the preceding example and ran into the notorious “world age” problem.

We feel like we are getting close to a solution but are not sure of how to move forward. We looked at Base.invokelatest() to address the problem, but not sure whether that is the solution nor to exactly implement it. Any thoughts? Thank you! ~tcp

1 Like
2

Requires.jl likes to be used in __init__, where it setups up a callback to run when a package is loaded to see if it was the one it wanted.

I think typical would be something like:

using Requires

function __init__()
    @require GTK="12345-2323..." begin
        using .GTK
        include("extra_file.jl")
    end
end

function main(call_extra=false)
    # ...
    if call_extra
        if isdefined(@__MODULE__, :GTK)
            extra()
        else
            error("GTK not loaded. To support extra mode first run using GTK, before calling main(true")
        end
    end
    #...
end

If you really want to do it in response to the argument i think the following work.

using Requires

function __init__()
    @require GTK="12345-2323..." begin
        using .GTK
        include("extra_file.jl")
    end
end

function main(call_extra=false)
    if call_extra && !isdefined(@__MODULE__, :GTK)
        @eval(:(using GTK))
        Base.invokelatest(inner_main, call_extra)
    else
        inner_main(call_extra)
    end
end

function inner_main(call_extra)
...
end

Might need to/want to use Base.requre instead of @eval(:using GTK))

Requires.jl doesn’t do great things to your net load time if it is needed though.
It causues the main task to block, when you load the package IIRC.

4 Likes
3

include ends up calling eval, so you are right, if you want to call extra in the same function its definition is included in, you need to make sure, you call it from the correct world age. In this case, just Base.invokelatest(extra) should do the trick.

4

(I think you meant Base.invokelatest(inner_main, call_extra) here)

2 Likes