Macros and modules

question
macros
metaprogramming
module

#1

Suppose I have a base package with abstract types and interfaces called BasePackage, and a second package with developer tools that builds on top of the base package called DevTools. In the latter package, I define helper macros to facilitate the creation of new solvers with similar interface:

DEVELOPER TOOLS

### BASE PACKAGE


module BasePackage
    abstract type AbstractSolver end
    abstract type AbstractEstimationSolver <: AbstractSolver end
    abstract type AbstractSimulationSolver <: AbstractSolver end

    export AbstractEstimationSolver, AbstractSimulationSolver
end # module


#------------------------------------------------------------


### DEVELOPER TOOLS


module DevTools

importall BasePackage
using Parameters: @with_kw_noshow

macro metasolver(solver, solvertype, body)
    solverparam = Symbol(solver,"Param")
    
    # discard any content that doesn't start with @param or @global
    content = filter(arg -> arg.head == :macrocall, body.args)
    
    # lines starting with @param refer to variable parameters
    vparams = filter(p -> p.args[1] == Symbol("@param"), content)
    vparams = map(p -> p.args[2], vparams)
    
    # lines starting with @global refer to global solver parameters
    gparams = filter(p -> p.args[1] == Symbol("@global"), content)
    gparams = map(p -> p.args[2], gparams)
    
    # add default value of `nothing` if necessary
    vparams = map(p -> p isa Symbol ? :($p = nothing) : p, vparams)
    gparams = map(p -> p isa Symbol ? :($p = nothing) : p, gparams)
    
    # replace Expr(:=, a, 2) by Expr(:kw, a, 2) for valid kw args
    gparams = map(p -> Expr(:kw, p.args...), gparams)
    
    # keyword names
    gkeys = map(p -> p.args[1], gparams)
    
    esc(quote
        @with_kw_noshow struct $solverparam
            $(vparams...)
        end
    
        struct $solver <: $solvertype
            params::Dict{Symbol,$solverparam}
            
            $(gkeys...)
    
            function $solver(params::Dict{Symbol,$solverparam}, $(gkeys...))
                new(params, $(gkeys...))
            end
        end

        function $solver(params...; $(gparams...))
            # build dictionary for inner constructor
            dict = Dict{Symbol,$solverparam}()

            # convert named tuples to solver parameters
            for (varname, varparams) in params
                kwargs = [k => v for (k,v) in zip(keys(varparams), varparams)]
                push!(dict, varname => $solverparam(; kwargs...))
            end

            $solver(dict, $(gkeys...))
        end
    end)
end

macro simsolver(solver, body)
    esc(quote
        @metasolver $solver AbstractSimulationSolver $body
    end)
end

macro estimsolver(solver, body)
    esc(quote
        @metasolver $solver AbstractEstimationSolver $body
    end)
end

export @estimsolver, @simsolver

end # module

USAGE

I expect contributors to use the packages as follows:

#### I AM A SOLVER DEVELOPER

importall BasePackage
using DevTools

@simsolver MySolver begin
    @param a = 2
    @param b
    @global gpu = false
end

The problem is that the exported macros defined inside of DevTools refer to names that are not exported like @metasolver. They also refer to names such as @with_kw_noshow that are defined inside of a third-party package Parameters.

I could start adding prefixes (e.g. Parameters) in front of every unqualified name, but that doesn’t seem right. How do you workaround this issue to provide such interface to external contributors?


Defining a solver type with macros
#2

esc the whole result is a bad style. You should only escape user input.

https://github.com/JuliaLang/julia/pull/22985 makes it almost impossible to call nested macros otherwise though.


#3

Thank you @yuyichao, how would you change the macros above to achieve this purpose? I am just starting with macros in Julia, and it is been a little difficult, at least for me, to grasp macro hygiene. I went over the docs a few times, but my attempts to fix the errors aren’t very educated.


#4

I am really stuck on this one, can someone give a hand?


#5

I think that if you didn’t escape everything, these unqualified names would get resolved correctly. But I wouldn’t know; I escape everything.

That’s what I do. Don’t use :(Parameters.@with_kw ...); use :($Parameters.@with_kw ...). The former will fail in modules that only import some of your constructs.


#6

Thank you @cstjean, it helped me understand the issue. I had to replace the line:

using Parameters: @with_kw_noshow

by

using Parameters # to make the symbol "Parameters" visible in the module

For future readers, this is the final solution:

### BASE PACKAGE


module BasePackage
    abstract type AbstractSolver end
    abstract type AbstractEstimationSolver <: AbstractSolver end
    abstract type AbstractSimulationSolver <: AbstractSolver end

    export AbstractEstimationSolver, AbstractSimulationSolver
end # module


#------------------------------------------------------------


### DEVELOPER TOOLS


module DevTools

importall BasePackage
using Parameters

macro metasolver(solver, solvertype, body)
    solverparam = Symbol(solver,"Param")
    
    # discard any content that doesn't start with @param or @global
    content = filter(arg -> arg.head == :macrocall, body.args)
    
    # lines starting with @param refer to variable parameters
    vparams = filter(p -> p.args[1] == Symbol("@param"), content)
    vparams = map(p -> p.args[2], vparams)
    
    # lines starting with @global refer to global solver parameters
    gparams = filter(p -> p.args[1] == Symbol("@global"), content)
    gparams = map(p -> p.args[2], gparams)
    
    # add default value of `nothing` if necessary
    vparams = map(p -> p isa Symbol ? :($p = nothing) : p, vparams)
    gparams = map(p -> p isa Symbol ? :($p = nothing) : p, gparams)
    
    # replace Expr(:=, a, 2) by Expr(:kw, a, 2) for valid kw args
    gparams = map(p -> Expr(:kw, p.args...), gparams)
    
    # keyword names
    gkeys = map(p -> p.args[1], gparams)
    
    esc(quote
        $Parameters.@with_kw_noshow struct $solverparam
            $(vparams...)
        end
    
        struct $solver <: $solvertype
            params::Dict{Symbol,$solverparam}
            
            $(gkeys...)
    
            function $solver(params::Dict{Symbol,$solverparam}, $(gkeys...))
                new(params, $(gkeys...))
            end
        end

        function $solver(params...; $(gparams...))
            # build dictionary for inner constructor
            dict = Dict{Symbol,$solverparam}()

            # convert named tuples to solver parameters
            for (varname, varparams) in params
                kwargs = [k => v for (k,v) in zip(keys(varparams), varparams)]
                push!(dict, varname => $solverparam(; kwargs...))
            end

            $solver(dict, $(gkeys...))
        end
    end)
end

macro simsolver(solver, body)
    esc(quote
        DevTools.@metasolver $solver AbstractSimulationSolver $body
    end)
end

macro estimsolver(solver, body)
    esc(quote
        DevTools.@metasolver $solver AbstractEstimationSolver $body
    end)
end

export @estimsolver, @simsolver

end # module


#------------------------------------------------------------


#### SOLVER DEVELOPER


importall BasePackage
using DevTools

@simsolver MySolver begin
    @param a = 2
    @param b
    @global gpu = false
end

It is a little convoluted, but it works.

@yuyichao could you please confirm that PR you linked won’t affect this macro in future releases of Julia?