Trouble understanding macros: creating a struct on the fly

macros
#1

I’m trying to create a macro that will help me build structs from a schema defined in a JSON document and while I have something that works, I would love a code review so I can learn how to do it better…

macro make_struct(struct_name, schema)
    fields=[:($(Symbol(entry[1])) :: $(Symbol(titlecase(entry[2])))) for entry in eval(schema)]
    return Expr(:type, false, eval(struct_name), Expr(:block, fields...))
end

And then I can verify how everything works here.

println(@macroexpand @make_struct :STRUCT_NAME [["x","int32"],["y","float32"]])
@make_struct :STRUCT_NAME [["x","int32"],["y","float32"]]
dump(STRUCT_NAME(1,2))

But my usage of eval and building my expression with the Expr(...) syntax feels like I’m missing a key point. Can anyone help me out?

#2 
macro make_struct(name, schema)
  fields=[:($(Symbol(entry[1]))::$(Symbol(titlecase(entry[2])))) for entry in eval(schema)]
  :(struct $(esc(name))
    $(map(esc,fields)...)
   end)
end
@make_struct STRUCT_NAME [["x","int32"]]

I’d say the thing you missed was adding the newline immediately after the struct name. It needs this for some reason.

Also you should probably escape the name of the struct and it’s fields otherwise the structs will be defined in the module the macro is defined in and it will look for DataType's used in the fields there too.

#3

The vanilla way of defining your macro is like this:

macro make_struct(struct_name, schema...)
    fields=[:($(entry.args[1])::$(entry.args[2])) for entry in schema]
    esc(quote struct $struct_name
        $(fields...)
        end
    end)
end
println(@macroexpand @make_struct STRUCT_NAME (x,int32) (y, float32))
@make_struct STRUCT_NAME (x, int32) (y, float32)

But my usage of eval and building my expression with the Expr(…) syntax feels like I’m missing a key point.

You should essentially never call eval inside a Macro. A macro is a source code transformation. It accepts code (Expr/Symbol objects), and outputs code, which will take the place of the macro invocation.

The Julia section on metaprogramming is pretty good. There are also beginner-level tutorials on Lisp macros on the web.

Once you’re more comfortable with macros, have a look at MacroTools. It makes writing them a lot easier.

#4

Thanks. I think the part I’m missing is how to go from my array of strings into the evaluated tuple of symbols.

My use case is I have a schema file S1.json on disk with the following structure [["x","int32"],["y","float32"]] and I’m trying to evaluate this into a struct so I can slurp in an array of these structs with open(f->read(f, S1, Int64(stat(filepath).size/sizeof(S1))), filepath, "r")

If I type out the fields in the structure, then this macro works just fine

function parse_schema(schema)
    function schema_helper(element)
        :($(Symbol(element[1]))::$(Symbol(titlecase(element[2]))))
    end
    tuple(map(schema_helper, schema)...)
end


z1=Any[Any["x","int32"],Any["y","float32"]] # from a JSON.parse(...)
println(parse_schema(z1))

macro make_struct(struct_name, schema...)
    esc(
        quote
            struct $(struct_name)
            $(schema...)
            end
        end
    )
end
println(@macroexpand @make_struct S1 parse_schema(z1))

I just get the following

begin
    struct S1
        parse_schema(z1)
    end
end

I understand that this is just parsing my expression and making a new one. Every time I think about how I would put a for loop on the inside of the struct, I come to the solution i should be writing a function that builds an Expr and then just evaling the new expression in my repl. Should I be avoiding a macro all together?

#5

Yeah, I think so. You can eval your structs into existence while parsing the file. You might run into world-age issues, though, but they are solvable with more eval.

Also consider that you can essentially define new data types at runtime with tuples and parametric types, without eval:

> struct JSONStruct{NAME, T<:Tuple}
    fields::T
    JSONStruct{NAME}(fields...) where NAME = new{NAME, typeof(fields)}(fields)
end

> JSONStruct{:apple}(1,2,"hey")
JSONStruct{:apple,Tuple{Int64,Int64,String}}((1, 2, "hey"))

> Apple = JSONStruct{:apple,Tuple{Int64,Int64,String}}

but you need to be very comfortable with parametric types, and it can get very messy. See eg. TypedTables.jl

#6

So I ended up writing the following functions to make my datatype and to slurp in my array of datatypes from disk.

function make_schema_struct_expr(dataset::String)
    function schema_helper(element)
        :($(Symbol(element[1]))::$(Symbol(titlecase(element[2]))))
    end
    schema=open(JSON.parse, "/path/to/data/$dataset/schema.json", "r")
    Expr(:type, false
        , Symbol(dataset)
        , Expr(:block, map(schema_helper, schema)...))
end
    
function load_dataset{T}(dataset::Type{T}, datafile_name)
    filepath = "/path/to/data/$dataset/$datafile_name"
    num_records::Integer = stat(filepath).size / sizeof(dataset)
    open(f->read(f, dataset, num_records), filepath, "r")
end

and then I just make sure I eval the struct and then load in my data.

eval(make_schema_struct_expr("MyDataset"))
dat = load_dataset(MyDataset, "file_0.dat")

I bet I can do this even more cleanly but this feels like a win. Thank you @cstjean for your help! =D

#7

Well, since the thread is titled “Trouble understanding macros…”, let’s use @cstjean’s first pattern:

module M
using JSON
export make_schema_struct

macro make_struct(struct_name, schema...)
    fields = [:($(x[1])::$(x[2])) for x in schema...]
    esc(quote struct $struct_name
        $(fields...)
        end
    end)
end

function make_schema_struct(datadir,dataset::String)
    schema = open(JSON.parse, joinpath(datadir,"$dataset/schema.json"), "r")
    schema = [(Symbol(x),Symbol(titlecase(y))) for (x,y) in schema]
    eval(current_module(),:(@M.make_struct $(Symbol(dataset)) $schema))
end
end

using M
make_schema_struct("/path/to/data","MyDataset")

Pretty much the same as yours, but a bit less expression-wrangling.

#8

Thanks @Ralph_Smith . This is great!