Macros and modules

General Usage
, , ,
1

Suppose I have a base package with abstract types and interfaces called BasePackage, and a second package with developer tools that builds on top of the base package called DevTools. In the latter package, I define helper macros to facilitate the creation of new solvers with similar interface:

DEVELOPER TOOLS

### BASE PACKAGE


module BasePackage
    abstract type AbstractSolver end
    abstract type AbstractEstimationSolver <: AbstractSolver end
    abstract type AbstractSimulationSolver <: AbstractSolver end

    export AbstractEstimationSolver, AbstractSimulationSolver
end # module


#------------------------------------------------------------


### DEVELOPER TOOLS


module DevTools

importall BasePackage
using Parameters: @with_kw_noshow

macro metasolver(solver, solvertype, body)
    solverparam = Symbol(solver,"Param")
    
    # discard any content that doesn't start with @param or @global
    content = filter(arg -> arg.head == :macrocall, body.args)
    
    # lines starting with @param refer to variable parameters
    vparams = filter(p -> p.args[1] == Symbol("@param"), content)
    vparams = map(p -> p.args[2], vparams)
    
    # lines starting with @global refer to global solver parameters
    gparams = filter(p -> p.args[1] == Symbol("@global"), content)
    gparams = map(p -> p.args[2], gparams)
    
    # add default value of `nothing` if necessary
    vparams = map(p -> p isa Symbol ? :($p = nothing) : p, vparams)
    gparams = map(p -> p isa Symbol ? :($p = nothing) : p, gparams)
    
    # replace Expr(:=, a, 2) by Expr(:kw, a, 2) for valid kw args
    gparams = map(p -> Expr(:kw, p.args...), gparams)
    
    # keyword names
    gkeys = map(p -> p.args[1], gparams)
    
    esc(quote
        @with_kw_noshow struct $solverparam
            $(vparams...)
        end
    
        struct $solver <: $solvertype
            params::Dict{Symbol,$solverparam}
            
            $(gkeys...)
    
            function $solver(params::Dict{Symbol,$solverparam}, $(gkeys...))
                new(params, $(gkeys...))
            end
        end

        function $solver(params...; $(gparams...))
            # build dictionary for inner constructor
            dict = Dict{Symbol,$solverparam}()

            # convert named tuples to solver parameters
            for (varname, varparams) in params
                kwargs = [k => v for (k,v) in zip(keys(varparams), varparams)]
                push!(dict, varname => $solverparam(; kwargs...))
            end

            $solver(dict, $(gkeys...))
        end
    end)
end

macro simsolver(solver, body)
    esc(quote
        @metasolver $solver AbstractSimulationSolver $body
    end)
end

macro estimsolver(solver, body)
    esc(quote
        @metasolver $solver AbstractEstimationSolver $body
    end)
end

export @estimsolver, @simsolver

end # module

USAGE

I expect contributors to use the packages as follows:

#### I AM A SOLVER DEVELOPER

importall BasePackage
using DevTools

@simsolver MySolver begin
    @param a = 2
    @param b
    @global gpu = false
end

The problem is that the exported macros defined inside of DevTools refer to names that are not exported like @metasolver. They also refer to names such as @with_kw_noshow that are defined inside of a third-party package Parameters.

I could start adding prefixes (e.g. Parameters) in front of every unqualified name, but that doesn’t seem right. How do you workaround this issue to provide such interface to external contributors?

2

esc the whole result is a bad style. You should only escape user input.

https://github.com/JuliaLang/julia/pull/22985 makes it almost impossible to call nested macros otherwise though.

1 Like
3

Thank you @yuyichao, how would you change the macros above to achieve this purpose? I am just starting with macros in Julia, and it is been a little difficult, at least for me, to grasp macro hygiene. I went over the docs a few times, but my attempts to fix the errors aren’t very educated.

4

I am really stuck on this one, can someone give a hand?

5

I think that if you didn’t escape everything, these unqualified names would get resolved correctly. But I wouldn’t know; I escape everything.

That’s what I do. Don’t use :(Parameters.@with_kw ...); use :($Parameters.@with_kw ...). The former will fail in modules that only import some of your constructs.

1 Like
6

Thank you @cstjean, it helped me understand the issue. I had to replace the line:

using Parameters: @with_kw_noshow

by

using Parameters # to make the symbol "Parameters" visible in the module

For future readers, this is the final solution:

### BASE PACKAGE


module BasePackage
    abstract type AbstractSolver end
    abstract type AbstractEstimationSolver <: AbstractSolver end
    abstract type AbstractSimulationSolver <: AbstractSolver end

    export AbstractEstimationSolver, AbstractSimulationSolver
end # module


#------------------------------------------------------------


### DEVELOPER TOOLS


module DevTools

importall BasePackage
using Parameters

macro metasolver(solver, solvertype, body)
    solverparam = Symbol(solver,"Param")
    
    # discard any content that doesn't start with @param or @global
    content = filter(arg -> arg.head == :macrocall, body.args)
    
    # lines starting with @param refer to variable parameters
    vparams = filter(p -> p.args[1] == Symbol("@param"), content)
    vparams = map(p -> p.args[2], vparams)
    
    # lines starting with @global refer to global solver parameters
    gparams = filter(p -> p.args[1] == Symbol("@global"), content)
    gparams = map(p -> p.args[2], gparams)
    
    # add default value of `nothing` if necessary
    vparams = map(p -> p isa Symbol ? :($p = nothing) : p, vparams)
    gparams = map(p -> p isa Symbol ? :($p = nothing) : p, gparams)
    
    # replace Expr(:=, a, 2) by Expr(:kw, a, 2) for valid kw args
    gparams = map(p -> Expr(:kw, p.args...), gparams)
    
    # keyword names
    gkeys = map(p -> p.args[1], gparams)
    
    esc(quote
        $Parameters.@with_kw_noshow struct $solverparam
            $(vparams...)
        end
    
        struct $solver <: $solvertype
            params::Dict{Symbol,$solverparam}
            
            $(gkeys...)
    
            function $solver(params::Dict{Symbol,$solverparam}, $(gkeys...))
                new(params, $(gkeys...))
            end
        end

        function $solver(params...; $(gparams...))
            # build dictionary for inner constructor
            dict = Dict{Symbol,$solverparam}()

            # convert named tuples to solver parameters
            for (varname, varparams) in params
                kwargs = [k => v for (k,v) in zip(keys(varparams), varparams)]
                push!(dict, varname => $solverparam(; kwargs...))
            end

            $solver(dict, $(gkeys...))
        end
    end)
end

macro simsolver(solver, body)
    esc(quote
        DevTools.@metasolver $solver AbstractSimulationSolver $body
    end)
end

macro estimsolver(solver, body)
    esc(quote
        DevTools.@metasolver $solver AbstractEstimationSolver $body
    end)
end

export @estimsolver, @simsolver

end # module


#------------------------------------------------------------


#### SOLVER DEVELOPER


importall BasePackage
using DevTools

@simsolver MySolver begin
    @param a = 2
    @param b
    @global gpu = false
end

It is a little convoluted, but it works.

@yuyichao could you please confirm that PR you linked won’t affect this macro in future releases of Julia?