Metaprogramming: Obtain actual type from Symbol for field inheritance

General Usage
, ,
1

Hello,

I admit, having read How to get the type of a symbol in a macro? I realize that this may be harder than I thought, but perhaps if I describe what I want to achieve, a solution can be made for this specific case.

In Agents.jl we have a macro, that comes directly from this answer: Composition and inheritance: the Julian way - #5 by dfdx that allows us to create “agents” (mutable structs) that have some fields coming from some other structs. Like so:

# The following is within the `Agents` module:
macro agent(name, base, fields)
    base_type = Core.eval(@__MODULE__, base)
    base_fieldnames = fieldnames(base_type)
    base_types = [t for t in base_type.types]
    base_fields = [:($f::$T) for (f, T) in zip(base_fieldnames, base_types)]
    res = :(mutable struct $(esc(name)) <: AbstractAgent end)
    push!(res.args[end].args, base_fields...)
    push!(res.args[end].args, map(esc, fields.args)...)
    return res
end

and having e.g., also defined sometjhing like

# this is also within `Agents` module
mutable struct GraphAgent
    id::Int
    pos::Int
end

A user can do

using Agents
@agent MyAgent GraphAgent begin
    name::String
end

which would generate a mutable struct MyAgent with the additional name field that would also have the fields id, pos.

Now, because of the usage of @__MODULE__, this macro can only be used to “inherit” types that are defined within Agents. We want to extend this to allow using types defined outside Agents. My thinking is that this would be simple to do if the user also provides the module the types were defined at. My problem is, I do not know how to use multiple dispatch in a macro to actually avchieve that. My idea is to allow the second macro version:


@agent MySecondAgent MyAgent MyModule begin
    age::Int
end

that now would allow using any type MyAgent defined in module MyModule (which will typically be Main), and “inhereting” the fields of that type.

I’ve tried to achieve this with several different versions, but I always mess up in ways I don’t understand… I can post here my failed code versions if it helps :smiley:

2

Is it that you need the module where the macro is expanded? Any macro has __source__ and __module__ defined implicitly, so you can always just use these variables when it’s helpful. To see the origin of a type, you can also use parentmodule. Your use case might have other subtleties, but I’d guess there might be a way to avoid having the user specify this.

As for multiple dispatch… Macros take the AST, nothing more. So what it knows is just what Julia knows after parsing: most things are Exprs. The macro itself can’t dispatch on the type an Expr eventually resolves to. A solution is to pass the values you want to dispatch on to a helper function of some sort, and have that take care of dispatch.

Here’s a toy example:

macro foo(ex)
    :(_foo($ex))
end

_foo(n::Int) = n^2

_foo(s::String) = println("Hello ", s, "!")

Then you get

julia> @foo 3
9

julia> @foo "world"
Hello world!
3 Likes
3

I think this is basically where you’re going wrong. Don’t use eval in the body of a macro–it’s going to evaluate that symbol at the time when the macro is expanded, which isn’t actually a particularly useful time, and the thing you’re referring to may not even have a value at that time.

For example, here’s how easily we can break this macro without even involving modules at all:

julia> let
         mutable struct Foo
           x::Int
         end
         
         @agent Bar Foo begin
           name::String
         end
       end

ERROR: LoadError: UndefVarError: Foo not defined

Just wrapping the code in a let block is enough to cause the macro to be expanded before Foo is defined because, again, you must never use eval within the body of a macro.

Instead, your macro needs to produce an expression which does the work you want. For example, if your macro expanded to something like:

quote
  base_fields = get_base_field_names_and_types(base)
  @eval struct Child
      base_fields
      child_fields
  end
end

Then it would work in all circumstances, and you’d never have to think about @__MODULE__ ever again.

3 Likes
4

Thanks @rdeits . I understand your points. The problem I am now is that I do not know how to create this get_base_field_names_and_types from base. Notice that base is just a symbol. That’s exactly why I used @eval to transform the Symbol to an actual Type whose fields I can obtain.

5

Thanks @cscherrer now I know how to multiple dispatch macros :slight_smile:

1 Like
6

This isn’t answering your question, but direct field inheritance isn’t necessarily the “standard” Julia way to do this. Often, composition is preferred and then some access functions or a getproperty redefinition can provide the comfortable syntax you’re after:

struct A
  fld1::Int
end
struct B
  a::A
  fld2::Int
end

# Option 1: define access functions
# this is highly extensible but a little tedious
# the @forward macro from Lazy.jl or MacroTools.jl can make this more convenient
getfld1(a::A) = a.fld1
getfld1(b::B) = getfld1(b.a) # just forward to the field
getfld2(b::B) = b.fld2

# Option 2: redefine getproperty to forward b.fld1 to b.a.fld1
function Base.getproperty(b::B, f::Symbol)
  f == :fld1 && return b.a.fld1
  # also any other properties you want to forward
  return getfield(b, f)
end

# Option 2a: redefine getproperty to forward b.FIELD to b.a.FIELD if possible
# this function might need to be @generated for type stability
function Base.getproperty(b::B, f::Symbol)
  if in(f, propertynames(fieldtype(B, :a)))
    return getproperty(b.a, f)
  end
  return getfield(b, f)
end

# Option 2b: "automatic" property forwarding to the first matching field
# this is almost certainly type-unstable but could be redesigned as @generated
# there is also a slight risk of general chaos
function Base.getproperty(b::B, f::Symbol)
  for i in 1:fieldcount(B)
    if in(f, propertynames(fieldtype(B, i))
      return getproperty(getfield(B, i), f)
    end
  end
  return getfield(b, f)
end

I don’t have a Julia session handy so I didn’t have a chance to test the above. There may be some minor mistakes. Also, options 2a and 2b might have performance issues (type instability) unless they are @generated so they can resolve propertynames at compile time (and possibly unroll the loop in the latter). I didn’t try to write a @generated getproperty because I don’t trust myself to get it close to correct without being able to test it.

7

Thanks @mikmoore however for my application the “object oriented way” is a much, much more intuitive way to go than defining access functions for every field. I know the later is the “more Julian way”, but, sometimes, the object oriented approach is just straight out better, and this is one of these times. I need to remain within the structs otherwise my users will have to define dozens of functions each time they want to use a new agent type with Agents.jl, which is not justifiable if I am being honest.

8

Couldn’t you define those dozens of functions as defaults for an abstract supertype, or is this not compatible with @agents somehow? (I never used Agents.jl before so idk.) This isn’t like inheritance because concrete types don’t subtype each other, but this is how you get many types to share a method.

9

No, because I don’t know in advance what kind of properties the user would want to create for their agent.

10

I meant the user would design such an abstract supertype and what properties it should have, so they would be defining those functions. For example, if they know all their types will just be an extension of a struct A, then they will design an abstract superA with subtypes A, B, C, ... with default methods f(::SuperA) that work on A’s properties.

Of course this is very limited, and it’s not feasible if you want to inherit fields from multiple types. Even in languages with multiple inheritance, composition is recommended if you want to “inherit” fields because field name overlap and method resolution order is really messy. Rather than field-wise getfld1 methods, I would rather define a getA or getB to get to the contained type and use their methods, like f(getB(x)), with a default f(x) = f(getB(x)) if I needed f to work on any type containing B (Lazy.@forward basically does this in a limited way).

11

That’s actually the whole point–base is not just a symbol in the context of the code generated by the macro. You’re using eval too early (in the body of the macro), which is the whole problem. You have to avoid doing that.

For example, here’s a macro that defines MyNewType that just copies the fields of the given type:

macro copy_type(base)
 quote
   let
     fields = fieldnames($(esc(base)))
     expr = quote
       struct MyNewType
         $(fields...)
       end
     end
     eval(expr)
   end
 end
end

Note how the macro body never calls eval. The only place where we eval is inside the expression generated by the macro itself. That’s the key.

julia> let 
         struct Bar
           x
           y
           z
         end
         
         @copy_type Bar

         @show fieldnames(MyNewType)
       end

fieldnames(MyNewType) = (:x, :y, :z)
(:x, :y, :z)
3 Likes
12

Thanks a lot for the help. While I understand a lot more, I still don’t understand everything to actually solve the problem :frowning:

I’ve managed to continue from your code into a version that obtains the fields and their types. Step 2 now is to allow a custom name for the new struct, and then step 3 would be to allow additional fields to be incorporated… However, I can’t get pass step 2. Here is what I have:

macro copy_type(base, newname)
  quote
    let
      base_fieldnames = fieldnames($(esc(base)))
      base_types = [t for t in getproperty($(esc(base)), :types)]
      base_fields = [:($f::$T) for (f, T) in zip(base_fieldnames, base_types)]
      expr = quote
        # Somehow enter the actual symbol of `newname` here...
        struct newname
          $(base_fields...)
        end
      end
      eval(expr)
    end
  end
end

mutable struct Example
    id::Int
end

@copy_type Example Example2

The problem is, I couldn’t get it to create a struct named Example2 no matter what I tried. Initially I thought I could just do $newname, and that would interpolate the Symbol into the quote. This errors however. Then, I thought, I need to escape newname before the inner quote, and then interpolate it in the inner quote. So I did new_name = :($newname) before the inner quote, and then used $(new_name) in the inner quote. This also didn’t work however. At this point I realize that I am trying random combinations of $ and esc, but I lack the understand to make the conscious choice of the correct combination. I am honestly confused of why the initial idea of just typing $newname didn’t work, as when the macro starts newname is just a Symbol, so plain interpolation in the expression should “work”…

Perhaps the most confusing thing is that when using simply $(newname) in the inner quote, the error it throws is

ERROR: syntax: invalid type signature around  .. / line with `($newname)`

which confuses me even more. How is the type signature suddenly affected by this? When I print the expression I get:

\expr = quote
    #= c:\Users\datse\Desktop\test_inheritance.jl:74 =#
    struct newname
        #= c:\Users\datse\Desktop\test_inheritance.jl:75 =#
        id::Int64
    end
end

which seems legit code, even though “newname” isn’t the name of the type I want.

13

Yeah, things get weird with multiple layers of quoting. I think QuoteNode is the magic piece you’re missing. Here’s an example:

julia> macro foo(newname)
         quote
           let
             name = $(QuoteNode(newname))
             expr = quote
               struct $name
                 x
                 y
               end
             end
             eval(expr)
           end
         end
       end
@foo (macro with 1 method)

julia> @foo MyType

julia> @show fieldnames(MyType)
fieldnames(MyType) = (:x, :y)
2 Likes
14

EDIT: This doesn’t work if I define the below macro in a module and call it outside its module. It defines the new type in the module the macro is defined, not the module the macro is called at.

EDIT2: Solved that by using Core.eval($(__module__), expr) instead of eval(expr)

Thank you very much @rdeits . With your guidance I was able to succeed in what I wanted to do. I post below the full solution for reference. I realized yesterday night that QuoteNode was what I needed, because the docs said that that its useful in nested quotes. But I admit, I couldn’t in the end understand exactly how to use it. Maybe I the metaprogramming docs could use more examples. Oh well, in any case, here’s the final product:


macro copy_type(base_type, newname, extra_fields)
  # This macro was generated with the guidance of @rdeits on Discourse:
  # https://discourse.julialang.org/t/
  # metaprogramming-obtain-actual-type-from-symbol-for-field-inheritance/84912

  # We start with a quote. All macros return a quote to be evaluated
  quote
    let
      # Here we collect the field names and types from the base type
      # Because the base type already exists, we escape the symbols to obtain it
      base_fieldnames = fieldnames($(esc(base_type)))
      base_fieldtypes = [t for t in getproperty($(esc(base_type)), :types)]
      base_fields = [:($f::$T) for (f, T) in zip(base_fieldnames, base_fieldtypes)]
      # Then, we prime the additional name and fields into QuoteNodes
      # We have to do this to be able to interpolate them into an inner quote.
      name = $(QuoteNode(newname))
      additional_fields = $(QuoteNode(extra_fields.args))
      # Now we start an inner quote. This is because our macro needs to call `eval`
      # However, this should never happen inside the main body of a macro
      # There are several reasons for that, see the cited discussion at the top for more
      expr = quote
        struct $name
          $(base_fields...)
          $(additional_fields...)
        end
      end
      # @show expr # uncomment this to see that the final expression looks as desired
      eval(expr)
    end
  end
end

mutable struct Example
  id::Int
end

@copy_type Example Example5 begin
  x::Float64
  y::Float64
end
1 Like
15

Aaaah the gods really hate me I think. Everything worked perfectly until I did the “very last” step of adding the macro in a module, and also include an abstract type to subtype from. Then, the QuoteNode trick does not work any more. Here is an example:

module TestMacro
export @copy_type

macro copy_type(base_type, newname, supertype)
  quote
    let
      base_fieldnames = fieldnames($(esc(base_type)))
      base_fieldtypes = [t for t in getproperty($(esc(base_type)), :types)]
      base_fields = [:($f::$T) for (f, T) in zip(base_fieldnames, base_fieldtypes)]
      name = $(QuoteNode(newname))
      supertype_quoted = $(QuoteNode(supertype))

      expr = quote
        struct $name <: $supertype_quoted
          $(base_fields...)
        end
      end
      eval(expr)
    end
  end
end

end

using .TestMacro

abstract type Abstract end

mutable struct Example
    id::Int
end

@copy_type Example Example2 Abstract

that errors with ERROR: UndefVarError: Abstract not defined.

I also tried to escape the abstract type, since it has already been defined, by doing $$(esc(supertype)) in place of where I currently have $supertype_quoted. This led to something really weird: the code no longer errored, but, oddly, Example2 wasn’t defined after I’ve run the macro. I still don’t understand how this broke the macro and it doesn’t evaluate Example2 anymore.

EDIT: Actually, forget about the Subtyping business. If I put the macro in a module, then even the version I posted above in the previous reply, that works perfectly, does NOT define Example2. I think the macro only defines these types in the module it was defined. But I want the opposite: I want these types to be defined in the module the macro is called in.

16

Okay! Solved the issue of Example2 not being defined by adding Core.eval($(__module__), expr) instead of just eval(expr).

The solution for the subtyping then become the $$(esc(supertype)) !!!