ANN: Setfield.jl Yet another package for working with immutables

FWIW I build a master from two days ago and it does not work:

julia> struct S{X,Y}
           x::X
           y::Y
           S(;x::X=1,y::Y=2) where {X,Y} = new{X,Y}(x,y)
           end

julia> Test.@inferred S(x=:a, y="")
ERROR: return type S{Symbol,String} does not match inferred return type Any
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] top-level scope
1 Like

I was using https://julialangnightlies-s3.julialang.org/bin/linux/x64/julia-latest-linux64.tar.gz linked from https://julialang.org/downloads/nightlies.html

I’m not sure which one in the https://status.julialang.org/ was that.

It seems that’s what Travis is using. For example, this shows “Commit a237986bf9 (2018-02-20 19:08 UTC)” and the @inferred with just keyword-only constructor is broken:
https://travis-ci.org/tkf/Reconstructables.jl/jobs/344120039#L442

This didn’t handle it?

Apparently not. You think this is a bug?

Worth opening an issue to track it (if that hasn’t been done already).

1 Like
3 Likes

It was apparently quite easy to get “theoretical” type-stability from the keyword-only approach, with a similar @generated trick:

@generated function recon(thing; kwargs...)
    fields = [Expr(:kw, k, :(thing.$k)) for k in fieldnames(thing)]
    return quote
        constructor = $constructor_of(thing)
        constructor(; $(fields...), kwargs...)
    end
end

Of course, it’s only “theoretical” at this point since the constructor itself is not type-stable.

Possible benefits over lens/setproperty approach are that (1) you can do a “batch” update in which the constructor is called only once and (2) it generates only one (or two ) function for each struct instead of length(fieldnames(...)) functions.

I may need to turn constructor_of into a generated function for type-stability.

1 Like

I would have thought that with the kw-constructor you shouldn’t need a generated function for “theoretical” type stability.

Well, I don’t know if it is actually needed since I’ve never tried it with Julia with type-stable keyword-only constructor. I just thought the getfield & fieldnames combo in the following code would be hard for Julia to make it type-stable. (or not?)

function recon(thing; kwargs...)
    constructor = constructor_of(thing)
    defaults = [(n => getfield(thing, n)) for n in fieldnames(typeof(thing))]
    return constructor(; defaults..., kwargs...)
end

The type of array defaults becomes Array{Pair{Symbol}} (the “value” part is undetermined) for sufficiently complex struct. If the type of thing has type parameters that depend on the fields, the return type cannot be determined, right? Can you avoid it?

Also, in Julia 0.6 and 0.7, it seems that the type of defaults itself cannot be inferred:

using Base.Test

struct S{X,Y,Z}
    x::X
    y::Y
    z::Z

    S(x::X, y::Y, z::Z) where {X, Y, Z} = new{X, Y, Z}(x, y, z)
end

get_fields(thing) =
    [(n => getfield(thing, n)) for n in fieldnames(typeof(thing))]

s = S(1, 2.0, "3")
@show @inferred get_fields(s)  # fails

The most specific inference I can get out of variants of get_fields was when I used tuple and generator. But it still was not perfect.

There is a way to avoid @generated functions, have full type stability and no need of a keyword constructor. It can be adapted to both the Lens design and your design. I sketch for simplicity the case of only updating a single field.

macro register(T)
    M = current_module()
    fields = eval(M, :(fieldnames($T)))
    quote
        function Reconstructables.recon(thing::$T, field, val)
            new_fields = map($fields) do fld
                fld == field ? val : getfield(thing, field)
            end
            constructor_of($T)(new_fields...)
        end
    end
end

The price is that you need to add @register MyStruct to each struct you want to use this feature with. This is even possible for structs living in other modules.

Isn’t it generating the similar amount code as the @generated function version? How is it better?

Also, I want to have custom inner constructors. In that case, you need to generate keyword-only or positional-only constructors anyway for it to have some systematic mutation mechanism. So, “you don’t need keyword-only constructors” is not a valid appealing point for use-cases I have in mind.

1 Like

Good question. I don’t know if this is better then using @generated, as I don’t know what the problem with @generated is. If its only the amount of generated code, then I don’t think its any better. I even think there is no way to get the same performance with less generated code. At the end of the day each variant with perfect performance has to generate fully specialized @code_llvm. So I guess the total amount of generated code will always be the same for fully typestable variants.

In Parameters.jl I require that there is a positional inner constructor for all fields defined (either automatically via @with_kw or by the user). That does all the enforcing of constraints and other things you may want to do. All other inner/outer constructors then go through that one. I think this scheme should be able to cover almost all cases.

1 Like

Yeah I think this is good design. I usually do the same thing:

  1. Have one and only one “low level” inner constructor. All other constuctors must go through this one in the end. Its only job is to enforce invariants, it does nothing else. It is positional and its signature is the fields of the struct.

  2. Have whatever high level outer constructors you like.

I think that’s a source of bugs. If I understand correctly, that means users have to manually call the inner constructor from the outer constructor with positional arguments. If there are many structs with more than 5 fields (say) it would be terrifying to do any kind of refactoring touching the fields.

(OK. We need to do that already when calling new. But please don’t increase the burden.)

Obviously, it does not imply that @with_kw is a bad design since it also generates a keyword-only constructor as well.

If you let users to write a inner-constructor, it has to be usable by humans. Positional-only constructor is not for humans. Humans are good at referring to things by names, not positions. I think that’s the primary reason why we use structs instead of tuples.

Then what is required for packages like Reconstructables.jl and Setfield.jl are to generate constructor usable by programs. That means a keyword-only or positional-only constructor. They both have pros and cons.

Okay I was not suggesting to apply this manually to every type under the sun. That is indeed error prone and ugly and what not. Also I was not suggesting this constructor for human use in types with lots of fields.

I think most types should have 0-3 fields and no custom constructor at all. If I apply this pattern I usually use a macro.

You are right, we need a standard constructor for the machine, for packages like Reconstructables.jl and Setfield.jl to work.
I think the positional constructor has one killer advantage. Its the default. This means it automatically works for most types. Especially useful for types defined in packages you don’t own.

I agree that the positional constructor has a huge advantage in that it is the (only) default at the moment. But keyworded constructor may have language support in the future as well:

Anyway, now that constructor lookup can be customized via Setfield.constructor_of, I guess I can do whatever I want to do on top of Setfield.jl which is great!

I started liking lens-based approach! Now that lens can change type parameters, we have “differentiable lens”!

using ForwardDiff
using Parameters: @with_kw, @unpack
using Setfield: @lens, Lens, set, get
import Setfield

@with_kw struct Coordinate{X, Y, Z}
    x::X = 1.0
    y::Y = 1.0
    z::Z = 1.0
end

# Since set called via ForwardDiff would specify a dual number, we
# have to ignore type parameters:
Setfield.constructor_of(::Type{<: Coordinate}) = Coordinate

function f(c::Coordinate)
    @unpack x, y, z = c
    return x^2 + y^2 + z^2
end

derivative(f, at, wrt::Lens) =
    ForwardDiff.derivative(
        (x) -> f(set(wrt, at, x)),
        get(wrt, at))

@assert derivative(f, Coordinate(x=1.0), @lens _.x) ≈ 2
@assert derivative(f, Coordinate(x=2.0), @lens _.x) ≈ 4

This is actually super useful for building something like bifurcation analysis tool where you need a generic way to let user specify a few parameters in a differentiable manner.

2 Likes

I had still doubts, that allowing to change the type was a good idea, but this example convinces me. Nice!

Glad that you like the example :slight_smile: