How can an user load a package extension?

[K5Julien]

I have read Julia’s documentation about package extensions as well as this GitHub example and though I first thought it was suitable for my use case, it seems it isn’t.

I have in my package a mutable struct SolveParameters that is used by the user so he can declare what he wants to do.

i.e., he could run the following in the REPL:

julia> import RingStarProblems as RSP
julia> using JuMP
julia> pars = RSP.SolverParameters(
        solve_mod      = RSP.BranchBendersCut(),   # ILP or B&BC
[...] Some other parameters
        do_plot        = false,                    # plot_results (to debug)
[...] Some other parameters
       )

When using the paramter do_plot = false, everything is fine. However, I would like that the user could run do_plot = true and only when he does so:

module PlottingSol #
    if pars.do_plot
        using RingStarProblems, GraphPlots, Compose
    
       # Do some plotting and saving to .pdf
    end
end ### /!\ this generates errors as you can't have `using` inside a function!

I tried to put the module PlottingSol into an extension as mentioned in Julia documentation, i.e., my Project.toml contains, among other dependencies removed for brevity:

[weakdeps]
Compose = "a81c6b42-2e10-5240-aca2-a61377ecd94b"
GraphPlot = "a2cc645c-3eea-5389-862e-a155d0052231"

[extensions]
PlottingSol = ["Compose", "GraphPlot"]

[compat]
Compose = "0.8, 0.9"
GraphPlot = "0.4.1, 0.5, 0.6"

[test] # For testing
Compose = "a81c6b42-2e10-5240-aca2-a61377ecd94b"
GraphPlot = "a2cc645c-3eea-5389-862e-a155d0052231"

However, at this point, I do not understand how the user is supposed to ask for package PlottingSol with do_plot = true.

[abraemer]

Package extension should add methods to existing functions. So you should do something like:
In the main part of RingStarProblems:

....
function perform_plot end # create function but do not add methods

function solve(...)
    # ...
    if do_plot
        # will error if no methods are defined
        perform_plot(...)
    end
end

In the extension PlottingSol (I think PlottingExt or PlottingSolutionExt would be slightly more preferred as name) you define a method:

module PlottingSol
using RingStarProblems, GraphPlots, Compose

function RingStarProblems.perform_plot(...)
    # do the plotting here
end

[ufechner7]

But this answer does not explain when an extension is loaded and when not.

[K5Julien]

Indeed! Thank you @abraemer, I coded what you suggested and now have the error:

ERROR: MethodError: no method matching perform_plot(::SolverParameters, ::RingStarProblems.RRSPInstance, ::Symbol, ::RingStarProblems.BDtable, ::Bool)
Stacktrace:
 [1] main(pars::SolverParameters, instdataname::Tuple{Symbol, Tuple{String, Int64}}, optimizer::MathOptInterface.OptimizerWithAttributes, solutionchecker::Bool)
   @ RingStarProblems ~/Documents/Travail/Julia/RingStarProblems.jl/src/main.jl:126
 [2] rspoptimize(pars::SolverParameters, symbolinstance::Symbol, optimizer::MathOptInterface.OptimizerWithAttributes, solutionchecker::Bool)
   @ RingStarProblems ~/Documents/Travail/Julia/RingStarProblems.jl/src/main.jl:36
 [3] rspoptimize(pars::SolverParameters, symbolinstance::Symbol, optimizer::MathOptInterface.OptimizerWithAttributes)
   @ RingStarProblems ~/Documents/Travail/Julia/RingStarProblems.jl/src/main.jl:18
 [4] top-level scope

Maybe because I do not know when to load the extension, as mentioned by @ufechner7?

[DanielVandH]

For extensions usually the point is that the user should be the one to do using PlottingSol. I would take @abraemer’s solution and just throw an error telling the user to do using. You can’t do using in non-top level statements (unless there’s some way to safely hack it using @eval I guess…).

[abraemer]

@K5Julien

Maybe because I do not know when to load the extension, as mentioned by @ufechner7?

Sorry I thought that this is rather clear in the documentation you linked. First sentence

A package “extension” is a module that is automatically loaded when a specified set of other packages (its “extension dependencies”) are loaded in the current Julia session.

So you never load a package extension manually. Package extensions are loaded automagically if the user adds the corresponding packages to the environment. So in your case, the extension is loaded if and only if the user added both Compose.jl and GraphPlot.jl to the environment either as a direct dependency (i.e. using Pkg.add) or indirectly by Pkg.adding packages that depend on Compose.jl and GraphPlot.jl. Note that the users also needs to load the load these other packages themselves in the current session to make the extension available.

Your usecase seems to not reflect this. So a package extension in the Julian sense is probably not what you want. It sounds more like you should just make a second, full package, e.g. RingStarProblemsPlotting.jl, that includes the functionality.

[K5Julien]

Thank you for your reply
Do you mean include("ext/PlottingExt.jl") instead of using PlottingExt? Because the latter throws the error:

julia> using PlottingExt
ERROR: ArgumentError: Package PlottingExt not found in current path.
- Run `import Pkg; Pkg.add("PlottingExt")` to install the PlottingExt package.

I am sorry but how to use an extension is still unclear to me. I have carefully read the Julia’s documentation about it without succeeding to make it works for the moment.

[DanielVandH]

The extension will be automatically loaded when the user does using Compose, GraphPlot (I meant this instead of using PlottingSol in the previous reply). I would just throw an error telling the user to load in those two packages to have access to your package’s plotting functionality.

Or indeed if this is something you want to be automatically loaded when the user tries to plot, maybe an extension is not the way to go as @abraemer mentions.

[K5Julien]

Alright, everything works perfectly now! Thank you for all your replies

[aris]

I was playing around with an extension recently and I have some questions about this:

@abraemer

....
function perform_plot end # create function but do not add methods

function solve(...)
    # ...
    if do_plot
        # will error if no methods are defined
        perform_plot(...)
    end
end

For users of the package who are potentially unfamiliar with the nuances of julia, I was thinking the following;
Instead of defining the “perform_plot” functon without an argument, which would lead to some “method not matching” error when called, we could define

function perform_plot(::whatevertype)
    @warn("The function perform_plot is not available unless you load the x package by typing 'using x' " )
end

to make it more user-friendly.

Would that be sensible, or am I getting something wrong?
I was planning to go down some metaprogramming rabbit hole, defining all possible extension functions will all their potential argument combinations using some eval loops, but it sound like a very cheesy solution.
Is there a more elegant way to achieve the same perhaps?

[abraemer]

@aris

I was planning to go down some metaprogramming rabbit hole, defining all possible extension functions will all their potential argument combinations using some eval loops, but it sound like a very cheesy solution.

This does not sound like a good solution to me. I have two ideas how to improve the MethodError.

1. You can check whether an extension is available with Base.get_extension(parent::Module, extension::Symbol). So you could simply do a check before trying to call perform_plot and error appropriately. This check could perhaps be made more convenient with a macro e.g. @with_extension PlottingExt perform_plot(...)
2. There is Base.Experimental.register_error_hint which allows you tack an additional message onto the MethodError that occurs.

Edit: I am not sure whether one could simply define a method in the main module that warns and then overwrite that method in the extension. This could potentially lead to recompilation/break precompilation or just not work at all. I am not sure.

[aris]

Thanks a lot for the suggestions!

I think I prefer 1. , since 2. would still result in the code crashing, which is not good if there’s anything after calling the empty function.
Just to clarify, do we have any documentation on the functions/macro you recommend? I would like do a bit of homework before throwing these into my package.

@abraemer

I am not sure whether one could simply define a method in the main module that warns and then overwrite that method in the extension.

It seems to work okay for some toy examples I was messing around with. But perhaps things could go wrong in more complicated cases.

[abraemer]

@aris

Just to clarify, do we have any documentation on the functions/macro you recommend?

Base.get_extension is documented. The macro was a suggestion from my side to make it more convenient if you have a lot of places where you would write something like:

if isnothing(Base.get_extension(@__MODULE__, :Extension))
    @error "To use myfunction you need to ensure that the extension Extension is loaded"
end
myfunction(..)

[kellertuer]

That depends a bit, how 2 errors. you could do soomething like

if isdefined(Base.Experimental, :register_error_hint)
        Base.Experimental.register_error_hint(MethodError) do io, exc, argtypes, kwargs
            if exc.f === convex_bundle_method_subsolver
                print(
                    io,
                    "\nThe `convex_bundle_method_subsolver` has to be implemented. A default is available currently when loading QuadraticModels.jl and RipQP.jl. That is\n",
                )
                printstyled(io, "`using QuadraticModels, RipQP`"; color=:cyan)
            end
    end
end

(I might be biased, it is from my own package, see here)

Then it does crash but clearly reports what to load. For me the advantage is, that a MethodError is thrown, which itself also provides alternatives available (compared to a error you throw yourself) – but the message you see above also tells about what to load to get a working variant.