Local scope in macro

General Usage
1

I’m trying to figure out how to make variables defined in a let expression be visible in a hygienic macro which is evaluated in that let expression.

macro insertx(x)
   x
end

let y = 5
  @insertx(y)
end
>UndefVarError: y not defined.

Note, because programmers have to manually call gensym in Common Lisp, analogous code works:

* (defmacro insertx (x) x)
INSERTX

*(let ((y 5))
   (insertx y))
5

I can get this macro to work by wrapping x in esc:

macro insert_nonhygienicx(x)
    esc(x)
end

let y = 5
    @insert_nonhygienicx(y)
end
>> 5

in this toy example the lack of hygiene isn’t a problem, but with more complicated examples it is. For instance, if I try to create the Lisp nif macro (from Let over Lambda), I run into problems.

macro nif(expr, pos, zero, neg) #Three argument "if".
   if expr > 0
      esc(pos)
   elseif expr == 0
      esc(zero)
   else
      esc(neg)
   end
end

let x = "Positive!"
   @nif(3, x, "Zero", "Negative")
end
>> "Positive!"

I can use a local variable in the pos, zero and neg arguments of @nif. But if I want to use a local variable in the expr argument, I get an error:

let x = 2
   @nif(x, "Positive", "Zero", "Negative)
end
>>  LoadError: Method Error: no method matching isless(::Int64, ::Symbol)

Here, for type reasons, I cannot wrap expr in esc. I could wrap the whole body of the macro in esc(quote...end), but it’s not hard to expand this example to one in which I need to define variables inside the macro (say as a function of expr), while preserving hygiene. Perhaps there’s a way to manually use gensym myself to hid only the variables internal to the macro (and which I therefore don’t want to interact with the outside code), but I can’t figure it out.

1 Like
2

I think there might some misconception about what macros do. All macros do is programmatically construct Julia code you don’t want to type over and over again.

This construction of julia code happens before the actual values of the variables you are working with are known. In your second example, if expr > 0, the macro doesn’t know the value is greater than 0 at the time the macro is working.

On the other hand, you might understand this, but are missing some details for how to properly construct the expressions. You need to use quote and :(...) a lot.

Remember that a macro accepts an expression and returns an expression (for the most part). You are right that the scope is super confusing. My advice is to always put the code that constructs the expression in a separate function, such that you have

macro nif(expr, pos, zero, neg)
    esc(nif_helper(expr, pos, zero, neg))
end

function nif_helper(expr, pos, zero, neg)
    quote 
        if $expr > 0
            $pos 
         elseif $expr == 0 
            $zero
       end
    end
end

julia> @nif 0 "pos" "0" "neg"
"0"
1 Like
3

I think my examples were unhelpful, especially the later ones. How do I explicitly use gensym in the body of a macro? Julia usually does that automatically, but sometimes I need to explicitly call it inside an esc and use the resulting symbol programmatically, in the body of the macro. (Like I would in Common Lisp.)

In Common Lisp I can use existing local variables as arguments in hygienic macros, though I have to manually call gensym myself. In Julia, I either get hygienic macros or the ability to use existing local variables as arguments in macros. But not both. If I call esc on the body of the macro, I can use existing local variables as arguments in macros, but I get buggy, unhygienic macros. On the other hand, if I don’t call esc on the body of the macro, I get a hygienic macro, but I cannot use existing local variables in the arguments of the macro. I should be able to explicitly use the symbol generated through gensym, but I can’t figure out how to (and the manual doesn’t discuss it).

Changing the nif_helper function above just a little we get very buggy code:

function nif_helper(expr, pos, zero, neg)
    quote 
        p = $expr + 5
        if p > 0
            $pos 
         elseif p == 0 
            $zero
         else
            $neg
       end
    end
end
>>nif_helper

let p = "positive"
   @nif 5 p "Zero" "Negative"
end #Should return "Positive"
>> 10

The issue is that p was captured by the macro. To avoid that, I shouldn’t use p but some unrepeatable returned by gensym. But I can’t figure out how to assign the symbol returned by a gensym call. How do I do so? Or if that won’t work (because I’m inside an esc) how do I make a hygienic macro whose arguments can be existing local variables?

Sometimes, as in my examples in the OP, I can call esc on portions of the code. But I couldn’t figure out how to make it so all the arguments of the macro could be existing local variables without including portions of the macro body that may contain variables which should not be captured (and so should not be inside esc) inside esc.

4

One other comment:

If we tried to manually expand the macro from my first example in

let x = 2
   @nif(x, "Positive", "Zero", "Negative)
end

it would seem we would get:

let x = 2
   if x > 0
         "Positive"
      elseif x == 0
         "Zero"
      else
         "Negative"
      end
   end
end

Which is perfectly legible Julia code.

The issue is not that the macro doesn’t know if expr > 0 at the time the macro is working. It shouldn’t have to, and doesn’t try to. The problem is that it doesn’t expand it like that, but to something like:

let x = 2
   if Symbol("##x#266") > 0
         "Positive"
      elseif Symbol("##x#266") == 0
         "Zero"
      else
         "Negative"
      end
   end
end
5

@colinclout12
You practically never have to gensym anything by hand. @pdeffebach `s approach is generally recommended but does not work as expected when you need to introduce local variables, because the esc works over the whole expression.
What you want to do is only esc what needs to be evaluated in the calling scope. This should work for your modified nif + 5 , but feel free to point me to problems with this approach if you find any:

julia> macro nif_plus_five(expr, pos, zero, neg)
           return quote
               value = $(esc(expr)) + 5
               if value > 0
                   return $(esc(pos))
               elseif value == 0
                   return $(esc(zero))
               else
                   return $(esc(neg))
               end
           end
       end
@nif_plus_five (macro with 1 method)

julia> @nif_plus_five 3 "Positive" "Zero" "Negative"
"Positive"

julia> let x = "letted Positive"
        @nif_plus_five -2 x "zero" "neg"
       end
"letted Positive"

julia> let value = "This should not interefere with the output: Positive!"
           @nif_plus_five 5 value "Zero" "Neg"
       end
"This should not interefere with the output: Positive!"

julia> let value = "This should not interefere with the output: Positive!"
           @macroexpand @nif_plus_five 5 value "Zero" "Neg"
       end
quote
    #= REPL[5]:3 =#
    var"#5#value" = 5 + 5
    #= REPL[5]:4 =#
    if var"#5#value" > 0
        #= REPL[5]:5 =#
        return value
    elseif #= REPL[5]:6 =# var"#5#value" == 0
        #= REPL[5]:7 =#
        return "Zero"
    else
        #= REPL[5]:9 =#
        return "Neg"
    end
end

This automatically gensyms value because you assign to it, but you can also write local value when you assign to it the first time just to be sure.

If you still find you need to manually handle gensym names, you have to declare them outside of the quote expression you return, then you can use them like normal variables. First with liberally escaping and without gensym, to confirm that this is a dirty macro normally:

julia> macro nongensymmed(left, right)
           return esc(quote
               value = $(left) + $(right)
           end)
       end
@nongensymmed (macro with 1 method)

julia> let value = 2
           println("Value: $value")
           solution = @nongensymmed 3 5
           println("Solution: $solution")
           println("Value after macro: $value")
       end
Value: 2
Solution: 8
Value after macro: 8

And now with a gensym name for value (you don’t need to specify a name in the gensym, it’s only changing the base of the name). You declare it before quote and interpolate it where you need it as usual:

julia> macro gensymmed(left, right)
           value = gensym("value")
           return esc(quote
               $(value) = $(left) + $(right)
               @show $(value)
           end)
       end
@gensymmed (macro with 1 method)

julia> let value = 2
           println("Value: $value")
           solution = @gensymmed 3 5
           println("Solution: $solution")
           println("Value after macro: $value")
       end
Value: 2
var"##value#246" = 8
Solution: 8
Value after macro: 2
4 Likes
6

This example works for me.

julia> function nif_helper(expr, pos, zero, neg)
           quote 
               p = $expr + 5
               if p > 0
                   $pos 
                elseif p == 0 
                   $zero
                else
                   $neg
              end
           end
       end
nif_helper (generic function with 1 method)

julia> macro nif(expr, pos, zero, neg)
       esc(nif_helper(expr, pos, zero, neg))
       end
@nif (macro with 1 method)

julia> @nif 0 "pos" "zero" "neg"
"pos"

Maybe you over-wrote the function at some point? Try in a new julia session and see what happens.

7

As an aside, one reason to use gensym is to avoid evaluating an expression multiple times. If you use $expr multiple times, but expr is something like p() where we define p as

function p()
    sleep(10)
    4
end

then you will have to call p() multiple times and incur a performance cost. Assigning the output of p() to a gensymed variable at the start will prevent this problem.

8

Thanks! That’s very helpful, and similar to what I was trying at first (see my first post where all the arguments were wrapped in esc except for expr). What does the quote block introduce so your code works but the following code doesn’t work?

julia> macro nif_plus_five(expr, pos,zero,neg)
    obscure_name = esc(expr) + 5
    if obscure_name > 0
        esc(pos)
    elseif obscure_name == 0
        esc(zero)
    else
        esc(neg)
    end
end
@nif_plus_five (macro with 1 method)

julia> @nif_plus_five 3 "Positive" "Zero" "Negative"
> LoadError: MethodError: no method matching +(::Expr, ::Int64)
9

It works as long as the variable p isn’t used somewhere else in the code. This is a new Julia session:

julia> function nif_helper(expr, pos, zero, neg)
                  quote
                      p = $expr + 5
                      if p > 0
                          $pos
                       elseif p == 0
                          $zero
                       else
                          $neg
                     end
                  end
              end
nif_helper (generic function with 1 method)

julia> macro nif(expr, pos, zero, neg)
              esc(nif_helper(expr, pos, zero, neg))
              end
@nif (macro with 1 method)

julia> @nif 0 "pos" "zero" "neg"
"pos"

let p = "pos"
          @nif 0 p "zero" "neg"
          end #Should return "pos".
5

julia> @nif 0 "pos" "zero" "neg"
"pos"

(By “should return “pos”” I don’t mean Julia is doing something wrong. It’s doing exactly what I told it to, including assigning p to 5. But good luck remembering that’s the problem if you put nif in that let block.)

10

True, you were almost there ^^
To answer that, consider what the macro does:

A macro runs like a normal function. Only it’s input and output are special.

The macro call in the function is replaced with the returned expression from the macro at compile time. So, when you call a function with a macro the first time, the function gets compiled and when the compiler encounters the macro, the macro is run and its out is pasted into the function and compiled with it. When you run the function again (with the same signature), it’s already compiled and the macro doesn’t run again.

The quote block (or :(...) ) constructs an expression that can be evaluated later. So in my version, the macro call is replaced by the whole if... elseif... else block. In you version, the compiler tries to run the whole if... elseif... else block at compile time and paste the chosen output into the function. Or it just fails if it can’t do that, like you saw.

I can show you better later, not at the pc at the moment.

Try to read the section on metaprogramming again and play around with @macroexpand and something like

macro showmewhatyousee(input)
    @show input
    @show typeof(input)
    input
end

It took me at least 4 reads to properly understand what :() , $() and esc are doing.

2 Likes
11

Yup, this is a textbook example of why to use gensym. You can’t pollute the namespace with commonly used names inside a macro.

12

Thanks! This is super helpful!

quote...end and :() seem to be Common Lisp’s backtick (`) or apostrophe ('), and $ to be Common Lisp’s , (I’m new there so I don’t understand these super well in Common Lisp), but esc doesn’t seem to have an analogue in Common Lisp, where programmers have to maintain macro hygiene themselves by using gensym manually, whereas in Julia the default is that the macro is hygienic, and any parts you don’t want to be hygienic need to be wrapped in esc. I’ll have to look up esc in the manual, but my impression now is that it tells the Julia compiler not to clean up the hygiene.

I think the biggest issue tripping me up is understanding precisely what precisely the computer does when it compiles code. I think of it as “turning a Haskell text file into a .exe, or a LaTeX text file into a .pdf.” But that’s obviously very superficial.

13

I think what I was missing is that a macro doesn’t just plop the text inside the macro into the code, invisibly manipulating the text file (is this what a C++ macro does?), but runs (some of) the code at parse time. (Or something like that?) So in my first post, it tries to run the if statement at parse time, before the let block runs. But when we wrap macro code inside a quote...end block, it evaluates the quote block into code at parse time, but only executes the code inside the block at run-time. Is that right?

2 Likes
14

Yes. exactly.

You can manipulate the expression with julia functions that run before “run-time”, but you can’t use any information like whether or not a variable is above or below 0. You can just use the actual content of the expression.

1 Like