Adding Hidden Fields to Parametric Types

General Usage
1

Say I want to define a macro which adds a hidden field to a user defined type. See the minimal working example below

module Foo 

""" Modify ex by pushing hidden field. """
function push_hidded_field!(ex)
    paramex = ex.args[2]
    bodyex = ex.args[3]
    push!(bodyex.args, :(y::S))
    push!(paramex.args, :S) 
    ex
end

macro deftype(ex)
    push_hidded_field!(ex)
    quote 
        $ex 
    end |> esc
end

export @deftype

end

Now consider some of the use case of this macro.

using .Foo 

# Initial definiton is not a problem. 
@deftype struct Bar{T} 
    x::T 
end

There is nothing wrong in this case.

julia> dump(Bar) 
UnionAll
  var: TypeVar
    name: Symbol T
    lb: Union{}
    ub: Any
  body: UnionAll
    var: TypeVar
      name: Symbol S
      lb: Union{}
      ub: Any
    body: Bar{T,S} <: Any
      x::T
      y::S

julia> fieldnames(Bar) 
(:x, :y)

Bar was defined as expexted with a hidden fieldname y. The problem occurs when I try to parameterize x with S.

julia> @deftype struct Bar{S} 
           x::S 
       end
ERROR: syntax: function static parameter names not unique around /tmp/deleteme4.jl:17
Stacktrace:
 [1] top-level scope at REPL[7]:1
 [2] include_string(::Function, ::Module, ::String, ::String) at ./loading.jl:1088

The complaint is that the names are not unique. This means that the user calling the macro cannot parametrize its fieldname x with S. I think this is restrictive and needs a better approach.

To solve this problem, let us consider that we parameterize our hidden field name y with gensym so that we avoid duplicate names. So let us redefine the macro

module Foo 

""" Modify ex by pushing hidden field. """
function push_hidded_field!(ex)
    paramex = ex.args[2]
    bodyex = ex.args[3]
    S = gensym() # Hidden parameter type be different with those of the user. 
    push!(bodyex.args, :(y::$S))
    push!(paramex.args, S) 
    ex
end

""" Define the macro """
macro deftype(ex)
    push_hidded_field!(ex)
    quote 
        $ex 
    end |> esc
end

export @deftype

end

and let us try some use cases.

julia> using .Foo 

julia> @deftype struct Bar{T} 
           x::T 
       end

julia> dump(Bar) 
UnionAll
  var: TypeVar
    name: Symbol T
    lb: Union{}
    ub: Any
  body: UnionAll
    var: TypeVar
      name: Symbol S
      lb: Union{}
      ub: Any
    body: Bar{T,S} <: Any
      x::T
      y::S

julia> fieldnames(Bar)
(:x, :y)

There is nothing wrong in the first call. The problem occurs when the macro is called multiple times in the same julia session.

julia> @deftype struct Bar{T} 
           x::T 
       end
ERROR: invalid redefinition of constant Bar
Stacktrace:
 [1] top-level scope at REPL[1]:16
 [2] include_string(::Function, ::Module, ::String, ::String) at ./loading.jl:1088

This time the problem is that after being defined Bar is constant and cannot be redefined.

Any suggestions about how to get rid of these restrictions?

2

That’s exactly the problem: types can not be re-defined. This hasn’t anything to do with your code, but is rather a design choice of Julia (which has been discussed numerous times, especially in the context of Revise; in case you’re interested in the reasons behind this, searching for these keywords should lead you to relevant discourse threads)

Apart from the context of interactive development, during which you might want to update the definition of an existing type (which is the Revise scenario), is there a reason why you’d want to have two successive definitions of the same type in your code?

2 Likes
3

Let us assume that the macro is defined such that the fieldnames except default values such as (as in the case of Base.@kwdef)

using .Foo 

@deftype struct Bar{T}
    x::T = t -> t  + 1. 
end

When this call is executed, Bar is defined. The default is called in the run time. Let us say I realize that I defined the default value of x wrong and want to correct it

@deftype struct Bar{T}
    x::T = t -> t  + 10. 
end

I cannot do this because I cannot redefine Bar

4

I may be missing something, but I am not sure in what sense you consider y “hidden”, when in fact it is like any other field, just part of the type definition is generated by a macro.

If you just want to add the same set of fields to various types, I would suggest putting them in a stuct and composing.

1 Like
5

There is nothing special about the word hidden, We may call it as additional or extra. I just want to add additional fieldnames to a type that the user does not provide.

Of course, composing types may be a solution. But I wondered if there is a possible solution using macros.

6

Yes you can:

julia> Base.@kwdef struct B{T}
       a::T = 1
       end

julia> B()
B{Int64}(1)

julia> Base.@kwdef struct B{T}
       a::T = 11
       end

julia> B()
B{Int64}(11)

because this does not re-define the type but only the keyword constructor method which fills in the 1 or 11.

1 Like
7

What I meant was that I cannot do this using @deftype macro, not using Base.@kwdef.

And, if I change the symbol of the type I end up with the same error.

julia> Base.@kwdef struct B{S}
       a::S = 11
       end
ERROR: invalid redefinition of constant B
Stacktrace:
 [1] top-level scope at util.jl:459
 [2] include_string(::Function, ::Module, ::String, ::String) at ./loading.jl:1088

So I think Base.@kwdef also tries to redefine the type. At least what I see from this line is this. https://github.com/JuliaLang/julia/blob/41781b1cb01fa8d2f387a1452de8e13b5295c08f/base/util.jl#L463

8

AFAIU the only way a type redefinition is allowed is if the new definition is exactly the same as the old one. This is the case with Base.@kwdef when only default values are changed (because default values are handled via specific constructors, not in the type definition).

But it is not the case with your macro as it’s currently written, since the name of the extra type parameter changes each time the macro expands (it gets gensymmed anew)

A possible workaround could be to always use the same gensymmed symbol for all types extended this way. I think this is legit because the only hygiene issue you want to avoid would be a clash between the extra type parameter name and any user-provided type parameter.

To expand on your initial example (I made it so Foo.@deftype implicitly forwards the type definition to Base.@kwdef so that default values can easily get defined):

module Foo
export @deftype

# Gensymmed only once, used for all type definitions
const EXTRA_FIELD_TYPE = gensym()

function push_hidded_field!(ex)
    paramex = ex.args[2]
    bodyex = ex.args[3]
    # Let's put a default value for the extra field as well
    push!(bodyex.args, :(y::$EXTRA_FIELD_TYPE = 0))
    push!(paramex.args, EXTRA_FIELD_TYPE)
    ex
end

macro deftype(ex)
    push_hidded_field!(ex)
    quote
        Base.@kwdef $ex
    end |> esc
end

end

This behaves how (I think) you’d expect:

julia> using .Foo

julia> @deftype struct Bar{T}
           x::T = 42
       end

julia> Bar()
Bar{Int64,Int64}(42, 0)

julia> @deftype struct Bar{T}
           x::T = 43.0
       end

julia> Bar()
Bar{Float64,Int64}(43.0, 0)
2 Likes
9

That was exactly what I was looking for. Thank your @ffevotte .