Scope in metaprogramming

General Usage
1

I am trying to define a function that delegates a task to a set of internal, very similar subfunctions. These subfunctions need access to names in the scope of the defining function.
Sounds like a job for metaprogramming, but I have a hard time making it work. Here is an MWE:

# It works in global scope
    for func in [:foo, :bar]
        @eval function $(Symbol(string(func, "_func")))()
            A[1] += 1
        end
    end

    A = [0]
    foo_func()
    bar_func()
    A  # result 2

#But not if I wrap it in a function

function test()
    for func in [:foo, :bar]
        @eval function $(Symbol(string(func, "_func")))()
            A[1] += 1
        end
    end

    A = [0]
    foo_func()
    bar_func()
    A
end
test() # result 0

This is clearly because @eval evaluates to the global scope. So, I must not be approaching this the right way. What is the recommended approach for achieving this?

Thanks!

2

Have you tried using a macro?

The following might not be a good macro, but it does show that macros can be used to translate/generate code at parse time, avoiding the scoping issues of eval.

macro def_funcs(prefixes...)
    funcdefs = Expr[]
    for prefix in prefixes
        funcname = Symbol(string(prefix, "_func"))
        funcdef = :(
            function $(esc(funcname))()
                A[1] += 1
            end
        )
        push!(funcdefs, funcdef)
    end
    return Expr(:block, funcdefs...)
end

Use macroexpand() to see what it’s doing:

macroexpand(quote @def_funcs foo bar end)
3

You can splice in the values you want but note that you won’t be able to call the function within the test() on 0.6.

4

That’s good to know - what is the suggested approach? I got my toy example to work with a macro based on @greg_plowman s:

macro def_funcs(prefix)
    funcname = Symbol(string(prefix, "_func"))
    quote
        function $(esc(funcname))()
            A[1] += 1
        end
    end
end 

function test()
   A = [1]
   @def_func foo
   @def_func bar
   foo_func()
   bar_func()
   A
end

But in the case of my real code it isn’t looking pretty, and I have a hard time getting this to look readable. Is a macro not the best approach?

Writing the code out in full:

function add_dependents!(pkg, vtnum)
    deps = dependents(pkg)
    for dep in deps
        if in(dep, dones)
            vt = findfirst(dones, dep)
            add_edge!(g, vt, vtnum)
        else
            j[1] += 1
            push!(dones, dep)
            add_vertex!(g)
            add_edge!(g, vtnum, j[1])
            add_dependents!(dep, j[1])
        end
    end
end


function add_dependencies!(pkg, vtnum)
    deps = dependencies(pkg)
    for dep in deps
        if in(dep, dones)
            vt = findfirst(dones, dep)
            add_edge!(g, vtnum, vt)
        else
            j[1] += 1
            push!(dones, dep)
            add_vertex!(g)
            add_edge!(g, j[1], vtnum)
            add_dependencies!(dep, j[1])
        end
    end
end

My current macrobased approach:

macro depfunc(func)
    funcname = Symbol(string("add_", func, "!"))
    add_edge1 = string(func) == "dependencies" ? :(add_edge!(esc(:(g)), vt, vtnum)) : :(add_edge!(esc(:(g)), vtnum, vt))
    add_edge2 = string(func) == "dependencies" ? :(add_edge!(esc(:(g)), vtnum, esc(:(j))[1])) : :(add_edge!(esc(:(g)), esc(:(j))[1], vtnum))
    
    return quote
        function $(esc(funcname))(pkg, vtnum)
            deps = $(esc(func))(pkg)
            for dep in deps
                if in(dep, esc(:(dones)))
                    vt = findfirst(esc(:(dones)), dep)
                    $add_edge1
                else
                    j[1] += 1
                    push!(esc(:(dones)), dep)
                    add_vertex!(esc(:(g)))
                    $add_edge2
                    $(esc(funcname))(dep, esc(:(j))[1])
                end
            end
        end
    end
end
5

Do your functions really need individual names? Just put them in an array or something.

module Wev

function test()
   fun = [() -> (A[k] += d) for (k,d) in enumerate([3,-1,4])]
   A = [0,0,0]
   fun[1]() ; fun[1]()
   fun[2]() ; fun[2]()
   fun[3]() ; fun[3]()
   A
end

@show test()

end
6

Put them in an array will be very slow. Doesn’t matter though if you won’t be calling this function many times.

It’s still unclear what exactly do you want to achieve. I assume you want to save some typing? There are various ways to clean up your macro a little bit (don’t use string, :f instead of :(f), generating an anonymous function instead) but if the goal is just to generate two similar functions this is about as good as you can do since the template in the macro is about as short as it can be.

7

How come? Putting the functions into an array doesn’t prevent type inferability.

8

Is it the anonymous functions that would be slow, or just storing them in an array? That is, is the following also very slow?

module Wev

function test()
   mkfun(k,d) = () -> (A[k] += d)
   foo_func = mkfun(1,3)
   bar_func = mkfun(2,-1)
   baz_func = mkfun(3,4)
   A = [0,0,0]
   foo_func() ; foo_func()
   bar_func() ; bar_func()
   baz_func() ; baz_func()
   A
end

@show test()

end
9

Yes, exactly. I saw I was defining almost identical functions twice, and thought it could be a good study for metaprogramming. I think the reason this seems clunky is that it needs access to variables defined in the calling scope, perhaps.
The code still fails, though, but it is probably hard to see why in this somewhat messy example.

10

Depending on how they are generated, it will since they can all have different types. I doubt the case the OP want can be expressed with a simple array comprehension.

You are not consistent on what you escape and what not. j for example is espaced a few times and used directly once. You need to make sure every use of captured variables are escaped. It’s probably cleaner to generate functions that takes more argument in global scope and create closures that captures some of the arguments in the local scope.

1 Like
11

Yes that little j was tricky. I think the need for control of scoping makes it safer to abstain from the macros here. But thanks for looking at it and giving advice!

12

The functions are recursive. I need the function’s name to let it call itself. But the names are not the problem, the scoping and escaping of the variables is. The recursion is also why I need access to variables in the scope where the functions are defined.

13

Could you pass those variables as arguments to the function?
Then use eval to define functions in global scope.

14

Yes. I just thought then that the macro-style implementation would end up being less elegant than naively just typing out the code like on my first try. This ideom of using recursion while keeping track in a global is something I use quite a bit for code that traverses networks. So I wondered whether there was a way to do this with metaprogramming. I haven’t looked into how well closures would work.
Thanks for your advice!

15

Here’s what it might like with just closures, more or less. I’ve probably managed to swap some parameters or something, but the scope issues and the recursion should be handled. I highlighted the most relevant parameter names with all caps. I pass dones from the calling scope and leave g and j in the definition scope; adapt as needed. I don’t know how fast or slow it may be in Julia.

module X
g, j = ...
function make_adder(GET, ADD!, dones)
    function add_deps!(pkg, vtnum)
         deps = GET(pkg)
         for dep in deps
             if in(dep, dones)
                 vt = findfirst(dones, dep)
                 ADD!(vtnum, vt)
             else
                 j[1] += 1
                 push!(dones, dep)
                 add_vertex!(g)
                 ADD!(j[1], vtnum)
                 add_deps!(dep, j[1])
             end
         end
    end
end
let dones = []
    add_edge1(m, n) = add_edge!(g, m, n) # or the other
    add_edge2(m, n) = add_edge!(g, n, m) # way around
    add_dependencies! = make_adder(dependencies, add_edge1, dones)
    add_dependents! = make_adder(dependents, add_edge2, dones)
    ...
    add_dependencies!(pkg, vtnum)
    add_dependents!(pkg, vtnum)
    ...
end
end
1 Like
16

Hey this is great! I am so surprised the best solution did not require metaprogramming. Thanks!

1 Like
17

In case anyone was curious, the code was for a recipe to plot the dependency structure of julia packages. These plots are quite interesting (focal package red, packages depending on that are green, dependencies of the focal package (and by extension also all the green packages) are blue).

789Ă—477 113 KB

6 Likes
18

And the final code. https://gist.github.com/mkborregaard/6a97365594bc333cb01da1d95ad36543