Surface AST to surface AST lowering/desugaring?

General Usage
,
1

Is there any function/package to do surface AST to surface AST lowering? I’m aware of Meta.lower but it does not produce a surface AST that can be used as an output of a macro.

The only relevant thing I know is @c42f’s WIP on moving lowering from flisp to Julia (https://github.com/JuliaLang/julia/pull/32201). But it sounds like it’s going to take sometime for all of the things listed to happen.

An example usage I have in mind is macro @~ of LazyArrays.jl. It can handle @~ vcat(A, B) but not @~ [A; B] at the moment. I’ve been thinking that it’d be easy to support this kind of fancy syntax if there is a way to lower expression [A; B] to vcat(A, B) etc.

I’ll probably write a subset of lowering myself if there is no out-of-the-box solution but it’d be nice to know if there is already a solution. I’m also wondering if it makes sense to use Cassette and operate on the IR level to do something like LazyArrays.@~.

2 Likes
2

I’m using MLStyle.jl to do something along these lines for my package. I have a function canonical that takes an Expr and returns another, and has rewrite rules like this:

:($f($(args...))) => begin
    rf = r(f)
    rx = map(r,args)
    :($rf($(rx...)))
end

The first line in the function is r=canonical, in order for all the recursions to work out.

3

Yeah, using something like MLStyle.jl and MacroTools.jl is useful when manipulating AST. But I was looking for more out-of-the-box solution.

To demonstrate what I need, I cooked up a package that does what I need GroundEffects.jl but it likely does not cover many cases.

julia> GroundEffects.lower(:[A B; C D])
:((hvcat)((2, 2), A, B, C, D))

julia> GroundEffects.lower(:[A B])
:((hcat)(A, B))

julia> GroundEffects.lower(:(y .+= f(x)))
:((Base.Broadcast.materialize!)(y, (Base.Broadcast.broadcasted)(+, y, f(x))))

julia> GroundEffects.lower(:(g(y) .+= f(x)))
quote
    var"##lhs#402" = g(y)
    (Base.Broadcast.materialize!)(var"##lhs#402", (Base.Broadcast.broadcasted)(+, var"##lhs#402", f(x)))
end
4

I’m not aware of anything which does this, but it’s an interesting question: can we have an “initial desugaring” pass which doesn’t introduce any of the lowered AST forms and can be used by macros which don’t want to deal with some surface syntax complexities?

The tricky thing is that correct desugaring almost always introduces some intermediate lowered forms. For example, [a;b] is lowered to Base.vcat(a,b) where the Base.vcat is actually GlobalRef(Base, :vcat) rather than the surface syntax :(Base.vcat).

2 Likes
5

ATM my solution is to just directly put function object Base.vcat in Expr. I don’t know if that’s a valid strategy in Julia core but I am not aiming that high. Is Base.vcat a good approximation of GlobalRef(Base, :vcat)? Can it produce incorrect result? Maybe in a baremodule?

6

I don’t know the definitive story for how GlobalRef is used in the implementation. GlobalRefs are a “partially symbolic” reference to a given name within a known module instance. If a const binding for that name exists in the module during inference it’s resolved immediately. Otherwise the binding is looked up at runtime.

For your use case I think inserting the value of Base.vcat eagerly into the AST should be fine.

1 Like
7

I see, actual lowering has to handle non-const global.

8

Yes that’s one reason why GlobalRef exists, but I think there’s some other benefits I don’t really appreciate (serialization maybe?)

1 Like