I have a Julia expression that references the variable x, and need to replace all x references to y. However, I don’t find a reasonable way to do this.
MacroTools is promising:
julia> MacroTools.replace(:( x + 1 ), :x, :y)
:(y + 1)
but still wrong:
julia> MacroTools.replace(:( f(x, (x=1,)) ), :x, :y)
:(f(y, (y = 1,)))
# should be :(f(y, (x = 1,)))
Any suggestions?
MacroTools.replace looks like an uncodumented internal. The right way to do this is with MacroTools.postwalk.
julia> using MacroTools
julia> MacroTools.postwalk(:( f(x, (x=1,)) )) do ex
ex == :x ? :y : ex
end
:(f(y, (y = 1,)))
The result is the same: it replaces occurrences of the x symbol even when they are unrelated to the x variable.
Oh, I misread the code you had posted, and confused the # should be with the result.
Can you explain what you mean here by “unrelated to the x variable”? Expressions don’t label objects by scope, they’re just symbols, the x on it’s own and the x in (x = 1,) are on the exact same footing as far as the Expr is concerned.
When running this code, Julia distinguishes between the two xs - so it’s possible to distinguish them. Clearly, only the first x in f(x, (x=1,)) is related to the x variable, and the second x is just a name of a field.
In a macro, I want to take an input expression depending on x, like the body in f(x) = ... body ..., and turn it into an equivalent expression that depends on y. Only variable references need to change, not field names.
Okay, so it sounds like you want the scopes of symbols in the expression to be resolved assuming the expression follows regular julia semantics.
When running this code, Julia distinguishes between the two
xs - so it’s possible to distinguish them.
Sure, but that happens during lowering which is after macroexpansion. It’s not a property of the expression but a property of the IR. Unfortunately, Julia does not really expose it’s own scoping resolution to us in a form that we can use.
I’ve run into the same need as you before, and to my knowledge so far, the only real option is to use JuliaVariables.jl for this. Unfortunately, JuliaVariables.jl is a buggy, incomplete subset of the actual full rules, so this can’t really be relied upon for general code, but depending on your use-case it might be enough.
julia> using JuliaVariables
julia> let ex = :( f(x, (x=1,)))
ex |> simplify_ex |> solve_from_local!
end
:($(Expr(:scoped, (bounds = Var[], freevars = Var[], bound_inits = Symbol[]), :((@global f)(@global x, (x = 1,))))))
What’s important here is that now the variables are labelled by their scope. So f is known to come from outside the expression, and one of the xs is known to come from outside but another is local.
You could then do a postwalk over this ‘solved’ expression and only replace the @global x instead of regular x. But I’d also seriously consider another path because this package is not fully reliable.
Here’s one way to do it with MacroTools. The obvious problem is that it only works for this particular case; generalizing this approach is probably too painful.
julia> MacroTools.postwalk(:(f(x, (x=1,)))) do ex
if @capture(ex, f_(x_, (a_ = b_,)))
x === :x ? :($f(:y, ($a=$b,))) : ex
else
return ex
end
end
:(f(:y, (x = 1,)))
Sure, that will work for the case shown in the OP, but I (maybe wrongly?) suspected that @aplavin’s actual usecase is significantly more complicated than that
Wow, didn’t expect this to be that deep of an issue!
Indeed, I want such replacements for arbitrary input expression, not for some specific forms. Moreover, I would argue that’s a common need for many transformation macros. Typically, this issue is totally ignored by macro writers just because there is no clear solution.
Is it feasible to expose variable resolution functionality from Julia directly as some function in Base?
I think such a thing might not be very straightforward because as far as I’m aware, the scope resolution is not an isolated process. It’s kinda just mixed into the process for how julia lowers things to IR.
This is probably a deep issue in any language supporting macros. E.g., in Common Lisp such a functionality is known as a code walker and apparently it is not possible to write a fully portable code walker in ANSI Common Lisp (see for example Macroexpand-All: an example of a simple lisp code walker for some of the difficulties).
To illustrate the issue, let’s make a small macro that replaces every x by y (just for the sake of argument and including the same problem that you mentioned above):
macro xy(expr)
MacroTools.postwalk(expr) do ex
ex == :x ? :y : ex
end
end
Now, try to use it in a context involving another macro (again this macro is just for the sake of argument and in general, macros should not introduce unbound identifiers)
macro inc_x()
x = esc(:x)
:($x + 1)
end
and it will fail miserably
julia> @xy let x = 1; 2 * x end
2
julia> @xy let x = 1; @inc_x() end
ERROR: UndefVarError: x not defined
Thus, to properly replace every free occurrence of an identifier it is at least required to expand all macros within its body – as otherwise it is impossible to know in which context the identifier will appear. This requires substantial knowledge about the language, i.e., handling of environments, scoping rules etc. I’m not aware of any language were such rules are fully exposed to the user.
My question was about expression that don’t contain macros: either they didn’t in the first place, or were already expanded. Then, it’s possible to distinguish variables vs all other symbols without any extra information: it’s clearly possible by looking at the code in complete isolation.
In Julia, we have @macroexpand that expands all macros. Is it what the article talks about?
You still need a code walker (macroexpand-all was just an example and yes, macroexpansion is easier in Julia than Common Lisp, i.e., there are no local or symbol macros) which is aware of all special forms, e.g., let, function, if, while, named tuples etc., in order to prevent replacing (or expanding) symbols in certain contexts. I’m not aware of any such tool, let alone a full documentation of all special forms, i.e., syntactic constructs with special rules for lowering or evaluation.