Request for suggestions on emitting Julia source code from expressions, meta-programming or templates

Any thoughts on how best to emit Julia source code from expressions, meta-programming or templates?

Option 1: Ideally a macro dynamically generate structs and functions from some template, capturing those artifacts as expressions, then generate Julia source code. Generated code would be validated to work (since it comes from the Julia compiler) and not be fragile. Conceptually a good idea. Practically it’s not clear how to achieve, or if this is currently possible. Any thoughts?

Option 2: Manually generate a source file by composing strings. This is a valid approach but may be more fragile and require more testing, and seems less elegant.

Specifically, I need access to Google Cloud Platform (GCP) BigQuery service (among others). GoogleCloud.jl doesn’t support BigQuery and another library, BGQ.jl works through the command line relying on an installed Google SDK. I’m looking for an out-of-the-box GCP library that just works. Not seeing one, I’ll write one. Google emits all their Client SDK libraries for supported languages (not Julia) based on their machine readable specifications (Google API Discovery Service  |  Google Developers). So this is basically what is needed. Since the need is there, I’m assuming plenty of others would be helped by this exercise. An automated approach also opens up the opportunity to emit the entire GCP API surface at once. Maybe best an exercise by Google or JuliaCloud, but lacking their support, any suggestions?

1 Like

Assuming that you really need to generate code, definitely, definitely option 1, not option 2. Julia’s metaprogramming facilities are excellent, and you would lose all of that elegance and flexibility by trying to manually generate source code from strings. Essentially, in order to safely generate valid Julia code as strings, you would have to build up your own data structures and functions for representing that code. But Julia already provides all of that, so why not use it?

All you need to do to satisfy option 1 is:

  • Produce the relevant struct and function definitions by building up Exprs
  • Pass those definitions to eval
  • That’s it

For example, let’s say we want to generate a struct which has fields whose names are stored in a text file:

julia> read_text_file() = ["a", "b", "c"] # just a dummy implementation for testing
read_text_file (generic function with 1 method)

julia> expr = quote
         struct Foo
           $(Symbol.(read_text_file())...)
          end
       end
quote
    #= REPL[2]:2 =#
    struct Foo
        #= REPL[2]:3 =#
        a
        b
        c
    end
end

julia> eval(expr)

julia> f = Foo(1,2,3)
Foo(1, 2, 3)

Easy!

The way I usually approach metaprogramming tasks like this is to first try to manually generate the kind of output I want and then use the dump() command to inspect that expression:

julia> target = quote
         struct Foo
           a
           b
           c
         end
       end
quote
    #= REPL[2]:2 =#
    struct Foo
        #= REPL[2]:3 =#
        a
        #= REPL[2]:4 =#
        b
        #= REPL[2]:5 =#
        c
    end
end

julia> dump(target)
Expr
  head: Symbol block
  args: Array{Any}((2,))
    1: LineNumberNode
      line: Int64 2
      file: Symbol REPL[2]
    2: Expr
      head: Symbol struct
      args: Array{Any}((3,))
        1: Bool false
        2: Symbol Foo
        3: Expr
          head: Symbol block
          args: Array{Any}((6,))
            1: LineNumberNode
              line: Int64 3
              file: Symbol REPL[2]
            2: Symbol a
            3: LineNumberNode
              line: Int64 4
              file: Symbol REPL[2]
            4: Symbol b
            5: LineNumberNode
              line: Int64 5
              file: Symbol REPL[2]
            6: Symbol c

as long as you can produce an Expr that matches your target, you can be reasonably sure that it will work.

One of the best parts of Julia metaprogramming is that you can let the parser do some of the work of generating valid Exprs for you. For example, let’s generate an empty struct expression:

julia> expr = quote
         struct Foo
         end
       end
quote
    #= REPL[4]:2 =#
    struct Foo
        #= REPL[4]:3 =#
    end
end

julia> dump(expr)
Expr
  head: Symbol block
  args: Array{Any}((2,))
    1: LineNumberNode
      line: Int64 2
      file: Symbol REPL[4]
    2: Expr
      head: Symbol struct
      args: Array{Any}((3,))
        1: Bool false
        2: Symbol Foo
        3: Expr
          head: Symbol block
          args: Array{Any}((1,))
            1: LineNumberNode
              line: Int64 3
              file: Symbol REPL[4]

The quote command created a block expression, and inside that block is our struct expression. To add a field to the struct, we just need to modify the args field of that inner expression:

julia> push!(expr.args[2].args[3].args, :a)
2-element Array{Any,1}:
 :(#= REPL[4]:3 =#)
 :a                

julia> expr
quote
    #= REPL[4]:2 =#
    struct Foo
        #= REPL[4]:3 =#
        a
    end
end

and now we can evaluate the result to actually define that struct:

julia> eval(expr)

julia> f = Foo(1)
Foo(1)

and at no point did we ever have to do any kind of string concatenation!

8 Likes

So simple! Thank you!

A small function for removing compiler line comments:

julia> expr = quote
         struct Foo
           a
           b
           c
         end
       end
quote
    #= REPL[1]:2 =#
    struct Foo
        #= REPL[1]:3 =#
        a
        #= REPL[1]:4 =#
        b
        #= REPL[1]:5 =#
        c
    end
end

julia> # remove line comments
       function clean(expr)
         io = IOBuffer()
         print(io, expr)
         s = String(take!(io))
         replace(s, r"[ ]*#=[^#=]*=#\n"=>"")
       end
clean (generic function with 1 method)

julia> print(clean(expr))
begin
    struct Foo
        a
        b
        c
    end
end

MacroTools also helps a lot.
I like to write code as separate functions that perform “passes” over the expression, generally using postwalk (sometimes prewalk), and @capture for pattern matching.

It also provides lots of other helpful/convenience functions, such as…

julia> using MacroTools: striplines, @q

julia> expr1 = quote
                struct Foo
                  a
                  b
                  c
                end
              end
quote
    #= REPL[4]:2 =#
    struct Foo
        #= REPL[4]:3 =#
        a
        #= REPL[4]:4 =#
        b
        #= REPL[4]:5 =#
        c
    end
end

julia> striplines(expr1)
quote
    struct Foo
        a
        b
        c
    end
end

julia> expr2 = @q begin
                struct Foo
                  a
                  b
                  c
                end
              end
quote
    struct Foo
        a
        b
        c
    end
end

Your clean function unfortunately turns the expression into the string, while the above return expressions.
You can look at the source code (and the README) for ideas on how to implement similar functions.

Many of them are brilliantly simple; the functions demoed here:

rmlines(x) = x
function rmlines(x::Expr)
  # Do not strip the first argument to a macrocall, which is
  # required.
  if x.head == :macrocall && length(x.args) >= 2
    Expr(x.head, x.args[1:2]..., filter(x->!isline(x), x.args[3:end])...)
  else
    Expr(x.head, filter(x->!isline(x), x.args)...)
  end
end

striplines(ex) = prewalk(rmlines, ex)

macro q(ex)
  Expr(:quote, striplines(ex))
end
2 Likes

Perfect, exactly what I need! Thank you.