How to make Configurations.jl option fields extensible (abstract?)

General Usage
1

I’m using Configurations.jl quite extensively to run some simulations and it’s pretty amazing!

There is one scenario, however, which I’m not quite sure how to solve in an “elegant” way, so I just want to gather some thoughts from other people. Perhaps there is a completely different solution to my problem and I’m just not seeing it right now.

Problem

Suppose I have a core package which defines some parent option type MainOption. One of its fields can be of one of different sub-option types, say SubOption1 and SubOption2. This can be done straightforwardly by using a union of the subtypes similar to this

@option struct MainOption
    sub::Union{SubOption1, SubOption2}
end

which automatically checks if the input fits one of the two concrete sub-option types.

Now consider a second package, which extends the core package and adds new features (which are not required or desired in the core package). These features get their corresponding sub-option types (say SubOption3), but it’s not possible to use the MainOption from the core package directly with the new sub-option, since the type union was hard-coded.

The basic question is: How to design an option type which is extensible from an outside package in this way (i.e. allow for adding new sub option types for a particular field)?

Obvious Solutions (?)

To my understanding, using abstract types or parametric types (which subtype some abstract type) for the fields are the only ways to allow the field type of a struct to be extended by types which are not known at the time of the struct definition.

But neither abstract, nor parametric types work out-of-the-box for option types, since the methods to construct the objects expect some concrete types.

Another obvious solution would be to just duplicate all the code in the extension package and make its own MainOptionExt which contains all the concrete sub-option types known at that time. That introduces a lot of new code and maintenance burden though.

Current Approach

What I’m thinking about doing currently is this (there is also an example script below):

  • use an abstract type to group the sub-options and make this the field type of the main option
  • “dynamically” (i.e. at the time of construction, not definition) find all concrete subtypes of the abstract type when trying to construct a config object
  • customize from_dict to get the behavior of the union types

This is how it looks like in a simple example:

Current solution 
using Configurations

abstract type AbstractSubOption end

@option struct SubOption1 <: AbstractSubOption
    key1::Int
end

@option struct SubOption2 <: AbstractSubOption
    key2::Int
end

@option struct MainOption1
    sub::Union{SubOption1, SubOption2}
end

@option struct MainOption2
    sub::AbstractSubOption
end

d1 = Dict(
    "sub" => Dict(
        "key1" => 1
    )
)

from_dict(MainOption1, d1) # Will work (goes through all the union types)
# from_dict(MainOption2, d1) # Won't work out of the box

#####
# Convenience function to grab all "leaf" types of an abstract type
using InteractiveUtils

function find_all_leaf_types!(leaveTypes, type)
    for st in subtypes(type)
        if isconcretetype(st)
            push!(leaveTypes, st)
        else
            find_all_leaf_types!(leaveTypes, st)
        end
    end
    return nothing
end

function find_all_leaf_types(type)
    leaveTypes = Type[]
    if isconcretetype(type)
        push!(leaveTypes, type)
    else
        find_all_leaf_types!(leaveTypes, type)
    end
    return leaveTypes
end
#####

# With this, we can re-route the construction to 
function Configurations.from_dict(MainType::Type{MainOption2}, of::Configurations.OptionField{:sub}, SubType::Type{AbstractSubOption}, x)
    return Configurations.from_dict_union_type_dynamic(MainType, of, Union{find_all_leaf_types(SubType)...}, x)
end

from_dict(MainOption2, d1) # Will work now

@option struct SubOption3 <: AbstractSubOption
    key3::Int
end

d3 = Dict(
    "sub" => Dict(
        "key3" => 1
    )
)

# from_dict(MainOption1, d3) # Won't work, since the union was hard-coded in the option type
from_dict(MainOption2, d3) # Will still work, since the current type union is constructed

I’m wondering if this is really the easiest or most straightforward solution to my problem. And also if there are some fundamental downsides which I’m overlooking here (*). Perhaps my assumption about making structs accept new types are not correct?

I did see this post by the package author @Roger-luo , mentioning customization of the object type construction, but if I understood that correctly, it would come down to the same thing in the end. I would also need to hook in the function that finds the candidate concrete types from the given abstract types.

If there aren’t any general downsides, is there a chance that this could turn into a feature of Configurations.jl to be able to use abstract field types and keep option types extensible? I guess one argument against that would be the need to “crawl” all concrete subtypes, which could easily tank performance for really large abstract types…

(*) At this point I’m not worried about performance hits from having a config with abstract field type. The current approach could also be modified to use parametric types, but that makes the custom from_dict method a bit more verbose.

As always, any hints, tips, comments are greatly appreciated :slight_smile:

2

I actually thought about this before and as you said because Julia allows putting new subtypes under an abstract type it’s hard to know if the subtypes satisfy the option type interface requirements without running it. And putting an abstract type as field type violates performance tips. I personally usually just use Unions and define things by composition. Once you adopt the idea of inheriting by composition you will find supertypes are less useful, and traits or interfaces are more frequently used. So I later find myself not actually need that much subtyping in defining option types. (Or even just no supertype at all)

But back to your question, I think your solution is the best you can do right now. (And actually I think rust serde uses the same idea supporting traits by asking for the Serielizer trait). Option types are just a collection of interfaces so you don’t even have to let the macro generate it for you.

I’m glad you already figured this out! I don’t currently have plans for supporting this as a) haven’t seen that much use cases; b) I won’t have the bandwidth working on new features until next summer. But still, please feel welcome to open an issue about the proposal and your more detailed use cases. We can think about how this further. In terms of a builtin support of abstract types I’m not against it if user already defined this way, so if there’s a PR modify the code generator doing this I’m happy to review it. But also as you mentioned it might be better to think if we can support this as parametric types.

Hope this helps.

1 Like
3

I ran into something like this, and ended up duplicating the code mostly. I was afraid of performance penalties of looking up type descendants in real time.

Glad to see an alternative!

1 Like
4

Thanks for the detailed answer, that definitely helps! I was almost sure that you would have thought about this already, but I couldn’t dig up anything here or on Github after a quick search.

Traits sound quite useful and I definitely want to try and play around with them more in my own projects (I currently still think mostly in terms of sub-/supertypes, but it can be pretty restricting sometimes).

What I didn’t get from your answer is if there is a straightforward way to use traits with the option types (as they currently are defined in Configurations.jl)? At the end of the day, the field of a struct needs a (preferably concrete) type and at the moment I don’t know in my “core package” what data might be required for potentially new sub-options defined outside. So if I have to choose the type/data structure which the MainOption can hold (or should expect), I would have to go for something very generic and lose part of the advantage of using specific option types.

I’ll definitely have a look at the Rust library, thanks for the hint!

Will do! I’m still debating with myself if supporting abstract types is a good idea. Parametric types would surely be better performance-wise, but finding the actual concrete type to construct is a bit more involved (or might even be impossible). I’ll give it a bit more deliberation, but I’ll gladly open an issue for further discussion.

Thanks again for your response.

5

Nice to hear that I wasn’t the only one!

EDIT: In my case the configurations are only used in a one-time setup (which might be called frequently in some situations, but compared to the actual work it’s mostly negligible) and the abstract option types are relatively small, so I just went for it – so far without big performance problems.

6

I think from package perspective, supporting abstract type actually makes sense. Becaues it’s valid Julia code. It’s up to the user whether or not using it. (like myself don’t find it that always useful).

What I didn’t get from your answer is if there is a straightforward way to use traits with the option types (as they currently are defined in Configurations.jl)?

so Julia doesn’t really have traits, what I meant is the option types in Configurations is defined as a collection of interface functions:

conceptually all @option does is just read your type definition and generate these overloads for you. In cases like above, you don’t have to use the automated code generator if it gets too dumb or not performant enough (we do have some performance issues that I never had the chance to fix, so you can see Pluto partially uses its own version of from_dict)

For a parametric type, if the type parameter implements these interface, it will be treated as an option type, e.g

julia> @option struct Foo{T}
           x::T
       end

julia> @option struct Bar
           x::Int
           y::Float64
       end

julia> Foo(Bar(2, 3))
Foo{Bar}(Bar(2, 3.0))

julia> x = Foo(Bar(2, 3))
Foo{Bar}(Bar(2, 3.0))

julia> to_dict(x)
OrderedCollections.OrderedDict{String, Any} with 1 entry:
  "x" => OrderedDict{String, Any}("x"=>2, "y"=>3.0)

julia> from_dict(Foo{Bar}, to_dict(x))
Foo{Bar}(Bar(2, 3.0))

so all you need is to restrict this type parameter to be a subtype of something (or just let it be a random type parameter)