Metaprogramming: adding an arg to a function

General Usage
1

Julia 1.0.

Me and metaprogramming, I don’t know, I am real slow at this!

I want to write a macro that, given a function definition

function f(x,y)
do something
end

will generate

function f(store,x,y)
do something
do something with store
end

now “x,y” is an array of expressions, I do not see how to put that back into the expression I am generating. Here is my MNWE:


macro store(ex) # given a function, create the same function with the additional first argument "store"
        local name     = ex.args[1].args[1]            
        local arglist  = ex.args[1].args[2:end]     # NB: array of Expr  
        local body     = ex.args[2]
        out= quote
            function $(esc(name))(store,$arglist) # almost!  arg is an array, and that does not work!
                $(esc(body))
                # do something with "store" here, not relevant for MWE
            end
        end
        return out
end

@store function f(x)
    y = x
    for i = 1:3
        y = 3y
    end
    z = y+2
    return 2z
end

store()=nothing
f(store,3)
2

This seems to work:

macro store(ex) # given a function, create the same function with the additional first argument "store"
    name     = esc(ex.args[1].args[1])
    arglist  = esc.(ex.args[1].args[2:end])     # NB: array of Expr
    body     = esc(ex.args[2])
    out= quote
        function $(name)(store, $(arglist...)) # almost!  arg is an array, and that does not work!
            $(body)
            # do something with "store" here, not relevant for MWE
        end
    end
    return out
end

@store function f(x)
    y = x
    for i = 1:3
        y = 3y
    end
    z = y+2
    return 2z
end

f(nothing, 3)

Note the use of the ... inside the $-interpolation.

2 Likes
3

Excellent!

It took me time to notice the esc.() call, which is also quite important.

Thank you, good teacher!

1 Like
4

You’re welcome. (Also mark it as solved.)

5

OK, will keep “mark as solved” in mind. A continuation question. Now that things work, I package them in a module, and again, I get baffled, here is my MNWE.

__precompile__()
module InsiderMVE
    using Printf
    export spy, @store, store

    function spy(targets)
        vals=[]
        tags=[]
        function store(val,tag)
            if tag ∈ targets
                push!(vals,copy(val))
                push!(tags,tag)
            end
        end
        store(val)=store(val,:unspecified)
        function get()
            return (vals,tags)
        end
        return store,get
    end

    macro store(ex)
        if ex.head== :(=)                             # @store y = expr
            local var = ex.args[1]                    # :y
            return quote
                $(esc(ex))                            # y = expr
                store($(esc(var)), $(Meta.quot(var))) # store(y,:y)
            end
        elseif ex.head== :function                    # @store function f(x::float64,y) return x*y end
            foo  = esc( ex.args[1].args[1])            # :f
            arg  = esc.(ex.args[1].args[2:end])        # [:(x::float64),:y]
            body = esc( ex.args[2])
            out= quote
                function $foo(store,$(arg...))          # f(store,,) x::float64,y) return x*y end
                    $body
                end
            end
            return out
        end
    end
end #module spying

which I test with (Julia 1.0.1)

using InsiderMVE

@store function f(x)
    y = x
    for i = 1:3
        @store y = 3y
    end
    @store z = y+2
    return 2z
end

(s,results)=spy([:y,:z])
r=f(s,3)
(vals,tags)=results()
display(vals)
display(tags)

causing error

ERROR: LoadError: UndefVarError: store not defined
Stacktrace:
 [1] macro expansion at C:\Users\philippem\home\GIT\julia\modules_1_0\src\Insider.jl:59 [inlined]
 [2] macro expansion at C:\Users\philippem\home\GIT\julia\modules_1_0\test\test_insiderMVE.jl:7 [inl
ined]
 [3] f(::Function, ::Int64) at C:\Users\philippem\home\GIT\julia\modules_1_0\src\Insider.jl:68
 [4] top-level scope at none:0
in expression starting at C:\Users\philippem\home\GIT\julia\modules_1_0\test\test_insiderMVE.jl:14

Now, when “store” is undefined - in what scope, at what time?

: )

6

This happens during your call to f(s, 3), so that we know that macro expansion went well (at least did not error out). What happens here is that store seems to be undefined within the body of f.

As is often the case with macros, a good debugging method consists in checking that the macro expansion does what you want it to do.

In your case:

julia> @macroexpand @store function f(x)
           y = x
           for i = 1:3
               @store y = 3y
           end
           @store z = y+2
           return 2z
       end

# (output manually shortened by removing lines indications)
quote
    function f(#7#store, x)
        begin
            y = x
            for i = 1:3
                begin
                    y = 3y
                    (Main.store)(y, :y)
                end
            end
            begin
                z = y + 2
                (Main.store)(z, :z)
            end
            return 2z
        end
    end
end

What we see here is that the first argument of f is a specific symbol #7#store designed to avoid clashing with the rest of the code (as part of macro hygiene). Whereas calls to store(y, :y) and store(z, :z) refer to a (non-existing) Main.store symbol.

I’m trying to think of a good way to ensure that the same symbol is used at both places, but I’m currently not sure how to do this wile keeping your current approach where there are 3 calls to @store, separately instrumenting the function definition and some instructions in it…

7

To illustrate the problem I mention above, consider the following macro:

macro store(ex)
    store = :store_123456 #<-- WARNING: this is not hygienic!
    if ex.head== :(=)                             # @store y = expr
        local var = ex.args[1]                    # :y
        return quote
            $(esc(ex))                            # y = expr
            $(esc(store))($(esc(var)), $(Meta.quot(var))) # store(y,:y)
        end
    elseif ex.head== :function                    # @store function f(x::float64,y) return x*y end
        foo  = esc( ex.args[1].args[1])            # :f
        arg  = esc.(ex.args[1].args[2:end])        # [:(x::float64),:y]
        body = esc( ex.args[2])
        out= quote
            function $foo($(esc(store)),$(arg...))          # f(store,,) x::float64,y) return x*y end
                $body
            end
        end
        return out
    end
end

It ensures that the expansions of

@store y = 3y

and

@store function f()
   # ...
end

refer to the same symbol (store_123456). But this macro is not hygienic: if the code you write in the definition of f happens to refer to a symbol store_123456, then you have a collision between the code introduced by your macro and the original definition of f.

gensym is a standard way of generating symbols which will not conflict with other variables. But if you have 3 separate macro expansions, then you’ll get 3 separate symbols and you’ll be back to the current problem (although in a more hygienic way)…

8

This macro should do what you want, and still be hygienic.

macro store(ex)
    # Auxilliary function to recursively transform the body

    # Most AST nodes should probably left untouched, but maybe not all. Let's
    # explicitly list what we don't want to transform...
    aux(ex::Union{Number,LineNumberNode}) = ex
    # ... and error out if we've forgotten anything.
    aux(ex) = error("Oops, I forgot something: ", ex)

    # Symbols coming from "user" code should be escaped
    aux(s::Symbol) = esc(s)

    # Expressions are transformed
    aux(ex::Expr) = if ex.head == :macrocall && ex.args[1] == Symbol("@store")
        # If this is a macro call of the type:
        #
        #    @store VAR = EXPR
        #
        # we want to transform it to something like:
        #
        #    begin
        #      VAR = EXPR
        #      store(VAR, :VAR)
        #    end
        #
        ex.args[2] :: LineNumberNode
        assignment = ex.args[3]
        @assert assignment.head == :(=)
        var = assignment.args[1]
        quote
            $(aux(assignment))
            store($(esc(var)), $(Meta.quot(var)))
        end
    else
        # In the general case, expressions are just recursively transformed
        Expr(ex.head, aux.(ex.args)...)
    end


    # @store should be called on a function definition
    @assert ex.head == :function
    foo  = esc(ex.args[1].args[1])
    arg  = esc.(ex.args[1].args[2:end])
    body = ex.args[2]
    quote
        function $foo(store, $(arg...))
            $(aux(body))
        end
    end
end

In particular, for your test case, it expands in the following way, where you see that the introduced store symbol (named #311#store in this particular case) can not conflict with “user-written” code.

julia> @macroexpand @store function f(x)
           y = x
           for i = 1:3
               @store y = 3y
           end
           @store z = y+2
           return 2z
       end

# Output manually edited to remove LineNumberNode annotations
quote
    function f(#311#store, x)
        begin
            y = x
            for i = 1:3
                begin
                    y = 3y
                    #311#store(y, :y)
                end
            end
            begin
                z = y + 2
                #311#store(z, :z)
            end
            return 2z
        end
    end
end

In order to keep your original syntax, the inner @store macro calls are used as annotations during the expansion of the outer @store call. They are removed from the expansion, so that they don’t have a chance to get expanded themselves. To put it another way, although there seem to be 3 distinct macro calls in this example, only the outer expansion takes place: the two inner ones are removed before they can be expanded. I’m not sure whether this is a very Julian way of doing things…


In any case, using this macro, your initial example should now work. BTW, I don’t think this specific problem had anything to do with modules…

3 Likes
9

Aha, thank you for the insight: My fundamental problem is that I do code transformation through multiple macro calls, creating the problem with non-matching symbols.

I still have to read your last post, but my comment at this point in time is that my @store y = 3y should not generate a call to a function, but just leave a tag (somehow!), so that @store function... should do all the code generation (change to function header and calls to @store… in one go and with one consistent f-local name for function store.

10

Exactly. And that’s what the macro in my last post does. Except macro expansion goes “inwards”, i.e. @store y = 3y does not “leave a tag” for @store function to do its magic: rather, @store function is expanded first, and the mere presence of @store y = 3y is the “tag”.

1 Like
11

Your code makes sense and is very interesting to read.

A agree with your question “is this julian code”. The code relies on an assumption on Julia’s behaviour: that the outer macro call is expanded/allowed to generate first. Of course one can work around that - using a special assignment operator y := ... to require storage.

Merci beaucoup!