[ANN] Configurations.jl - Options/Configurations made easy

Package Announcements
1

I made a small package Configurations.jl to separate the utilities we wrote for Pluto.jl and Comonicon.jl for parsing configurations/options from TOML files and kwargs and serialize them back. (there are still 3 days left until it’s registered in General)

Why?

In quite a lot cases, we will want to read configurations from a TOML/JSON/YAML/XML file, such as Web applications, CLI applications, or just some ML research project that needs to specify a fairly large set of hyper-parameters. This is usually done via the following steps:

  1. parse the configurations:
    • it can be a file and parsed via a corresponding parser, e.g TOML.jl, YAML.jl JSON.jl etc. to a Dict
    • it can be kwargs from some function
  2. define a set of Julia structs as an intermediate representation to make further processing easier
  3. convert this Dict or kwargs object to the structs we just defined
  4. propagate these configurations struct instances to places use them
  5. some times, the configurations may get edited either interactively or programatically, we may need to serialize them back to Dict or a TOML file etc.

I personally find these steps repeats quite frequently in a lot packages I participate, and I kept creating similar things.

How to use

this package exports a macro called @option it is similar to Base.@kwdef but it defines some extra methods to help you create types for configurations, e.g we have two configurations types

using Configurations

@option struct Person
   name::String = "John"
   age::Int = 20
end

@option struct User
    person::Person = Person()
    tag::String = "none"
    is_something::Bool = false
end

first we want to read our configuration from a TOML file example.toml:

tag = "red"
is_something = true

[person]
name = "Roger"
age = 25

we can just call

from_toml(User, "example.toml")

on the other hand, you might still want to have a keyword argument interface for interactive usage,
you can just call from_kwargs to convert it from keyword arguments, the keyword arguments are auto-generated to handle hierarchical structures in a flattened manner:

from_kwargs(User; person_name = "Roger", person_age = 20, is_something = true, tag = "Blue")

In the end, you can serialize this configuration back to TOML, via to_toml or convert it back to OrderedDict via to_dict.

Have fun!

20 Likes
2

Thanks for this! Why do you need a special macro in the struct? Can’t you do this with any plain old struct?

1 Like
4

Yes you can - if you want to rewrite the same thing again and again. The point of making this package is to not write similar things in different places.

If you read the implementation in Pluto or Comonicon you will find it’s implemented by hand indeed but not necessary.

This macro just auto generate what you will and have to write by hand

5

What I meant is that doesn’t seem necessary to have a macro defining the struct : ie why can’t you do from_toml(any_random_struct, "example.toml") even if struct any_random_struct was not prefixed by @option? To implement from_toml it should be sufficient to use introspection routines like fieldnames

3 Likes
6

How does it compare to Preferences.jl?

2 Likes
7

Good point! This is because you still want to distinguish “normal” struct and “option” struct so that we can map hierarchical dictionary better e.g

The from_dict function is recursively called if there’s no such distinguish then it will convert the whole dictionary to some struct or convert the whole struct recursively to a dictionary which might not be what we want.

2 Likes
8

I think in general they aim for different things - Pereferences.jl aim for packages so it’s coupled with packages closely while Configurations is used for more general case. You can use it for packages, small scripts, etc and choose to use your own toml or other format or not or use it from CLI inputs

This is why we originally write these things in Pluto and Comonicon- we need a unified representation for options for CLI inputs, keywords argument configuration and maybe a TOML file and it’s not coupled with a certain package but some other project or script user wrote. And propagate this configuration struct to many different functions.

Actually I think you might also end up defining some of your own structs after you get the configuration from Preferences - it’s always convenient to pass only one argument that contains your configuration.

1 Like
9

To be short: if you want to config a package use Preference if you as a developer want to let your user configure some of your functionality using either TOML kwargs or other format (e.g CLI inputs) use Configurations.jl

But I do feel this is something should be in stdlib since Pkg created similar structs to handle Project.toml internally as well. Maybe it could be in Preference?

2 Likes
10

Couldn’t you also achieve that behavior by defining an Option abstract type to be a common supertype for those structs you want treated as options?

11

yes you can, this is back to the question of whether one should use traits or abstract types. In this case, I think traits are preferred since these options type can be used to dispatch other methods once it finishes reading configurations from kwargs or TOML files. Traits will let you be able to use your own abstract type.

When I implement these option types by hand, yes, I usually create an abstract type, but note this is a more general package to free people from doing this type of work, e.g you have some optimizers that are either gradient based ADAM or non-gradient based CMAES they may need to be dispatched to different interface since gradient based methods may need AD and non-gradient based methods may need other things.

You would want to declare ADAM <: GradientMethod and CMAES <: NonGradientMethod rather than make them both subtype of AbstractConfiguration - the property of configurable is not an abstract type, but a property that a lot things have, so I choose to not restrict this.

1 Like
12

I am currently developing a package,

https://github.com/MagneticParticleImaging/MPISimulations.jl

where I face a similar situation. I want to be able to parse structs from TOML files here. My biggest issue is, that the types in my structs are often parametric. E.g.

struct SinusoidalField{T,U:<StaticMagneticFields{T}}
    spatialFieldProfile::U
    frequency::T
end

has the field spatialFieldProfile which can be any subtype of the abstract type StaticMagneticFields{T} (e.g. StaticHomogeneousField{T}). So my approach has been to encode this information into the TOML file like so, which works

[SinusoidalField]
frequency = 2.5e6
[SinusoidalField.spatialFieldProfile.StaticHomogeneousField]
magneticFieldStrength = [0.12,0.12,0.12]

Can you think of a better solution?

A different issue I have are structs which have array and tuple like fields. How to handle the case, where e.g. I want to parse a struct like this on?

struct foo{N}
    bar::NTuple{N,ComplexF64}
end
13

Hi.

Thanks for this package.

I must be doing something wrong, but

str = “”"
name = “Testing”

[Optimization]
gamma_1 = 0.95
nouter = 10
ninner = 2000

“”"

fd = open(“config.toml”,“w”)
write(fd,str)
close(fd)

than>

@option struct Optimization
gamma_1::Float64 = 1.0
nouter::Int64 = 1
ninner::Int64 = 5000
end

@option struct Config

opt::Optimization=Optimization()

name::String=""

end

configuracoes = from_toml(Config,“config.toml”)

returns

Config(;
opt = Optimization(;
gamma_1 = 1.0,
nouter = 1,
ninner = 5000,
),
name = “Testing”,
)

where the field “name” is correct, but the fields under Optimization are still showing the default values.

If I use

TOML.parsefile(“config.toml”)
Dict{String,Any} with 2 entries:
“name” => “Testing”
“Optimization” => Dict{String,Any}(“gamma_1”=>0.95,“ninner”=>2000,“nouter”=>1…

it recovers the correct values from the config file.

What am I missing? Thanks!

14

It should be your field name rather than your type name, we won’t be able to distinguish different fields with the same type if we use type name do we? The corresponding TOML file should be

name = "Testing"

[opt]
gamma_1 = 0.95
nouter = 10
ninner = 2000
1 Like
15

type parameters are things you would like to know before creating this type, thus first it should appear somewhere in your TOML file or they should be inferred from your input type info.

there are cases when you have multiple similar option types for one field, e.g in a TOML file

[MyStruct.field.A]
propertyA = "A"
propertyB = "B"

will assign field with option type A and similarly you can have option type B for the same field, in this case, you will need to overload from_dict function for MyStruct (for current Configurations master branch) then use default_from_dict to convert the rest, then you will be able to use multiple types with your own syntax, e.g for this case

function Configurations.from_dict(::Type{<:MyStruct}, d::AbstractDict{String})
      type = first(keys(d["field"]))
      if type == "A"
          T = AOptions
      elseif type == "B"
          T = BOptions
      end
      MyStruct(default_from_dict(T, d["field"][type]), <other fields>)
end

A different issue I have are structs which have array and tuple like fields. How to handle the case, where e.g. I want to parse a struct like this on?

Tuples are not valid TOML fields, you should either not use them if you want to use TOML file or define your custom conversion rules for it.

for the multiple option type case I’m think about extending the macro syntax to free people from writing it manually. But it’s not there yet.

1 Like
16

I see. Thanks.

17

Some new features in the v0.3 version:

Option Type Alias

now you can declare an alias for your option type, e.g

@option "option_a" struct OptionA
   field::String = "some field"
end

@option "option_b" struct OptionB
   name::String = "some name"
end

@option struct MyOption
   other::Union{OptionA, OptionB}
end

and now from_toml function (or similarly fro from_dict) can create new instance of such structure by reading the following TOML file

[other.option_a]
field = "a field"

julia> from_toml(MyOption, "demo.toml")
MyOption(;
    other = OptionA(;
        field = "a field",
    ),
)

or it creates OptionB for

[other.option_b]
name = "a name"

julia> from_toml(MyOption, "demo.toml")
MyOption(;
    other = OptionB(;
        name = "a name",
    ),
)

Custom Type Conversion

Different formats support different built-in types and they could be uncompatible with Julia types. Now you can define this by overloading Configurations.option_convert, e.g

we can define how to convert String to VersionNumber

Configurations.option_convert(::Type{OptionB}, ::Type{VersionNumber}, x::String) = VersionNumber(x)

Default Value Reflection

it is sometimes very useful to be able to access the default value of each field, which Base.@kwdef does not support, but now @option supports it, one can access the default values via Configurations.field_defaults(::Type) as long as you defined your type using @option.

Some of the above features are only possible with macros, thus again, for whoever thought this is easier without macros, you may be able to implement what you need in a specific context, but in the more generic context, the macro @option is necessary. But I’m sure some of the efforts in this package could be merged into Base or Preferences since there are overlapping with Base.@kwdef and Preferences as well as some of the stdlibs.

2 Likes
19

Configurations v0.11

new documentation site available here:

https://rogerluo.me/Configurations.jl/dev/

Option Field Name Alias

https://rogerluo.me/Configurations.jl/dev/advance/#Field-Name-Alias

Sometimes, the struct field name is defined using a UTF-8 character such as Ω . It may cause incompatibility when converting to other format such as JSON/TOML or just a Dict for JSON-based REST API.

You can define an alias for the field in this case using the following syntax

@option struct MyType
    "alpha"
    α::Int
    "omega"
    Ω::Float64 = 1.0
end

Custom Option Macro

In some cases, you may not want all the features we defined by default in Configurations , such as the printing, etc.

In this case, you can construct your own macro using the code generation passes defined in Configurations . The code generation passes API starts with codegen_ .

Configurations uses an intermediate representation to represent user-defined option types, which is the OptionDef struct.

e.g you can modify the passes included easily:

function codegen(def::OptionDef)
    quote
        $(codegen_struct_def(def))
        Core.@__doc__ $(def.name)
        $(codegen_kw_fn(def))
        $(codegen_convert(def))
        $(codegen_show_text(def))
        $(codegen_is_option(def))
        $(codegen_field_default(def))
        $(codegen_field_alias(def))
        $(codegen_alias(def))
        $(codegen_isequal(def))
        $(codegen_show_toml_mime(def))
        $(codegen_create(def))
    end
end

function option_m(@nospecialize(ex), alias=nothing)
    def = OptionDef(ex, alias)

    quote
        $(codegen(def))
        nothing
    end
end
2 Likes
20

This is indeed a great software and thanks for developing it, it saved me many lines. I successfully used it to parseBool, Float and Int.

> But I was not successful to read Symbol into toml file, how do I do ?

@option struct DATA
      Kθ::Bool
      Pedological⍰::Symbol
   end # struct DATA

TOML FILES

[data]
    "Pedological⍰" = ":Core" # <:Core>; from traditional data set <:Smap> from Smap; <No> no data available
21

Symbol is not a TOML native type, thus you will need to define a type conversion specific to your option type defined above to convert the String to your target object, e.g VersionNumber is supported by default

22

Thanks, I am wandering if YAML format would be a better option as it seems to support symbols and Greek.

It will be great if you can help to fix from_dict which does not work.

# DEFINING STRUCTURE
using Configurations, YAML
@option  mutable struct EVAPOTRANSPIRATION
      α::Symbol
      Evaporation::Int64
   end

   @option  mutable struct SOIL
      Topsoil::Float64 
      Macropore::String 
   end

   @option  mutable struct OPTION
      evapotranspiration::EVAPOTRANSPIRATION
      soil::SOIL 
   end

function TOML_TEST()
   # PARSING TOML FILE
      Path= "D:\\Main\\MODELS\\SoilWater-ToolBox2\\src\\Temporary\\Toml.yaml"
      # TomlParse = TOML.parsefile(Path)

      YamlDict = YAML.load_file(Path)

      println(YamlDict)
# Dict{Any, Any}("evapotranspiration" => Dict{Any, Any}("Evaporation" => 1, "α" => ":Large"), "soil" => Dict{Any, Any}("Topsoil" => "Top", "Macropore" => 3))

      option = Configurations.from_dict(OPTION, YamlDict)
      # option = Configurations.from_toml(OPTION, Path)
  
   # TESTING
      println(option.evapotranspiration.Evaporation)
      println(option.evapotranspiration.α)
      println(option.soil.Macropore)
      println(option.soil.Topsoil)
end

The input file

# YAML INPUT FILE
evapotranspiration:
   α: :Large
   Evaporation: 1

soil:
   Macropore:  3
   Topsoil: "Top"

ERROR MESSAGE

ERROR: LoadError: MethodError: no method matching from_dict(::Type{OPTION}, ::Dict{Any, Any})
Closest candidates are:
  from_dict(::Type{T}, ::AbstractDict{String, V} where V; kw...) where T at C:\Users\pollaccoj\.julia\packages\Configurations\kq1sK\src\parse.jl:27```