Metaprogramming: how to use variables from enclosing scope?

General Usage
1

I want to use @eval to generate code that does the same thing for different symbols. I keep going back to this example from Logging.jl, even though it’s not the best example. Here is my contrived use case:

"""
find all elements of `x` and `y` according to some rule and put them into aptly named dictionaries. return the dictionaries.
"""
function func()
    x = rand(10)
    y = rand(10)

    x_idx = x.< 0.5
    y_idx = y.> 0.5

    for (gg,idx,vals) in ((:xDict,:x_idx,:x),(:yDict,:y_idx,:y))
        @eval begin
            ($gg) = Dict(zip($idx,$vals[$idx]))
        end
    end
    return (xDict,yDict)
end

julia> func()
ERROR: UndefVarError: x_idx not defined
 in eval(::Module, ::Any) at ./boot.jl:234
 in eval(::Module, ::Any) at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 in func() at ./REPL[84]:9

How can I use x_idx inside this @eval?

2

does this make it for you?

function func()
           x = rand(10)
           y = rand(10)

           x_idx = find(x.< 0.5)
           y_idx = find(y.> 0.5)

           for (gg,idx,vals) in ((:xDict,x_idx,x),(:yDict,y_idx,y))
               @eval begin
                   ($gg) = Dict(zip($idx,$vals[$idx]))
               end
           end
           return (xDict,yDict)
       end
3

yes! cool, thanks. bit stupid on my behalf. i think i got it now, actually much simpler than I thought. this is literally just interpolating symbols into expressions…

4

Note that is “solution” is not interpolating symbols into the expression. Also this is not the proper way to do metaprogramming since almost everything are running in the global scope (xDict and yDict are global variables). It is impossible to use eval to get or set local variables and it is extremely unlikely it will ever be possible.

5

I’m not sure I understand. my use case puts that example function inside a module like so:


julia> module Testm

       function func()
                  x = rand(10)
                  y = rand(10)

                  x_idx = find(x.< 0.5)
                  y_idx = find(y.> 0.5)

                  for (gg,idx,vals) in ((:xDict,x_idx,x),(:yDict,y_idx,y))
                      @eval begin
                          ($gg) = Dict(zip($idx,$vals[$idx]))
                      end
                  end
                  return (xDict,yDict)
              end

       end
Testm

julia> xDict = 0
0

julia> yDict = 0
0

julia> z=Testm.func()
(Dict(7=>0.282879,4=>0.471219,9=>0.10263,10=>0.24691,2=>0.257844,1=>0.378451),Dict(10=>0.76719,5=>0.543434,6=>0.858241))

julia> xDict
0

so it doesn’t seem to overwrite the global xDict. Would you recommend something different to achieve what I’m after?

6

A eval statement will always evaluate in the global scope of the module it is located in. Each module has its own separate glocal scope. Your second xDict sits in the global scope of Main whereas the one from the eval is in Testm:

julia> module Testm

              function func()
                         x = rand(10)
                         y = rand(10)

                         x_idx = find(x.< 0.5)
                         y_idx = find(y.> 0.5)

                         for (gg,idx,vals) in ((:xDict,x_idx,x),(:yDict,y_idx,y))
                             @eval begin
                                 ($gg) = Dict(zip($idx,$vals[$idx]))
                             end
                         end
                         return (xDict,yDict)
                     end

              end
Testm

julia> Testm.xDict
ERROR: UndefVarError: xDict not defined

julia> Testm.func() # this creates Testm.xDict
(Dict(7=>0.366333,4=>0.193313,3=>0.341151,5=>0.141716,8=>0.479359),Dict(3=>0.865234,5=>0.722114,8=>0.759312))

julia> Testm.xDict
Dict{Int64,Float64} with 5 entries:
  7 => 0.366333
  4 => 0.193313
  3 => 0.341151
  5 => 0.141716
  8 => 0.479359
7

ok i see - this may be an undesirable side effect indeed. thanks for pointing this out. any ideas for how to do this diffrently most welcome.

8

Do no use eval in functions. The Logging.jl code you referred to does not do that in a function and is meant to define globals.

If you want to save some typing on the gg =Dict(zip(idx, vals[idx])) line, you can write a macro, say @m, to expand @m gg idx vals to that. Sth like

macro m(gg, idx, vals)
    :($(esc(gg)) = Dict(zip($(esc(idx)), $(esc(:($vals[$idx]))))))
end

Should do it.

2 Likes
9

If saving typing is the actual goal, maybe you can take some idea from this:

xDict, yDict = map(((x_idx, x), (y_idx, y))) do t
                 idx, vals = t
                 Dict(zip(idx, vals[idx]))
               end
10

This is an oversimplification. One should not use eval unless one intends to have effects at global scope. But if one is building new types and methods specialized on those types, functions may be helpful. Graham (“On Lisp”) says to use functions rather than macros where possible. I suspect that some of the macro code in the spectacular StaticArrays package could be cleaned up or made more flexible by use of functions employing eval. Perhaps @andyferris has tried this and found otherwise?

11

Sure, but not by much. There are few cases that you need to do it but I don’t think any cases mentioned anywhere in this thread so far is one of those.

In most of those cases you shouldn’t put the code in a function to start with.

At a quick glance that’s exactly the opposite. This is also why I don’t want to use the few corner cases to confuse people that don’t need to worry about it. For most people, remember to not use eval in a function should be good enough.

2 Likes
12

I’m not certain how to use eval nicely in StaticArrays. The macros (e.g. v = @SVector [1,2,3]) are just there to convert common Julia array creation syntax into the appropriate StaticArrays constructor, because (a) the types have many type parameters and can be pain to write by hand and (b) people are familar with the standard array creation syntax. I could (and should) use functions inside my (disgusting) macros to amalgamate common code. Calling eval however would be very slow.

(as an aside, where I would love to use is in a @generated function’s generator but that is forbidden because the compiler writers (quite reasonably) don’t want to e.g. guarantee that function generators run exactly once, or deal with complications that mutating the state of the system in the middle of certain compilation steps would involve, which is fair enough! :slight_smile: )

But yes, in (rare) cases where you are doing a lot of complicated code generation or need to something special in __init__() then it might be reasonable to use eval() in functions that usually are called once (or a fixed number of times) when the package compiles or is loaded.