# \[ANN\] Configurations.jl - Options/Configurations made easy

**URL:** <https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669>\
**Category:** Package Announcements\
**Created:** [November 24, 2020, 5:12am UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669 "2020-11-24T05:12:20Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [November 24, 2020, 5:12am UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/1 "2020-11-24T05:12:21Z")

</div>

I made a small package [Configurations.jl](https://github.com/Roger-luo/Configurations.jl) to separate the utilities we wrote for [Pluto.jl](https://github.com/fonsp/Pluto.jl) and [Comonicon.jl](https://github.com/Roger-luo/Comonicon.jl) for parsing configurations/options from TOML files and kwargs and serialize them back. (there are still 3 days left until it’s registered in General)

## Why?

In quite a lot cases, we will want to read configurations from a TOML/JSON/YAML/XML file, such as Web applications, CLI applications, or just some ML research project that needs to specify a fairly large set of hyper-parameters. This is usually done via the following steps:

1. parse the configurations:
  - it can be a file and parsed via a corresponding parser, e.g TOML.jl, YAML.jl JSON.jl etc. to a `Dict`
  - it can be kwargs from some function

2. define a set of Julia `struct`s as an intermediate representation to make further processing easier
3. convert this `Dict` or kwargs object to the `struct`s we just defined
4. propagate these configurations `struct` instances to places use them
5. some times, the configurations may get edited either interactively or programatically, we may need to serialize them back to `Dict` or a TOML file etc.

I personally find these steps repeats quite frequently in a lot packages I participate, and I kept creating similar things.

## How to use

this package exports a macro called `@option` it is similar to `Base.@kwdef` but it defines some extra methods to help you create types for configurations, e.g we have two configurations types

```julia
using Configurations

@option struct Person
   name::String = "John"
   age::Int = 20
end

@option struct User
    person::Person = Person()
    tag::String = "none"
    is_something::Bool = false
end

```

first we want to read our configuration from a TOML file `example.toml`:

```toml
tag = "red"
is_something = true

[person]
name = "Roger"
age = 25

```

we can just call

```julia
from_toml(User, "example.toml")

```

on the other hand, you might still want to have a keyword argument interface for interactive usage,  
you can just call `from_kwargs` to convert it from keyword arguments, the keyword arguments are auto-generated to handle hierarchical structures in a flattened manner:

```julia
from_kwargs(User; person_name = "Roger", person_age = 20, is_something = true, tag = "Blue")

```

In the end, you can serialize this configuration back to TOML, via `to_toml` or convert it back to `OrderedDict` via `to_dict`.

Have fun!

---

<div class="post-metadata">

**Author:** ![antoine-levitt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antoine-levitt/32/4008_2.png) [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)\
**Post date:** [November 24, 2020, 7:33am UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/2 "2020-11-24T07:33:44Z")

</div>

Thanks for this! Why do you need a special macro in the struct? Can’t you do this with any plain old struct?

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [November 24, 2020, 2:13pm UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/4 "2020-11-24T14:13:51Z")

</div>

Yes you can - if you want to rewrite the same thing again and again. The point of making this package is to not write similar things in different places.

If you read the implementation in Pluto or Comonicon you will find it’s implemented by hand indeed but not necessary.

This macro just auto generate what you will and have to write by hand

---

<div class="post-metadata">

**Author:** ![antoine-levitt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antoine-levitt/32/4008_2.png) [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)\
**Post date:** [November 24, 2020, 2:23pm UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/5 "2020-11-24T14:23:24Z")

</div>

What I meant is that doesn’t seem necessary to have a macro defining the struct : ie why can’t you do `from_toml(any_random_struct, "example.toml")` even if `struct any_random_struct` was not prefixed by `@option`? To implement `from_toml` it should be sufficient to use introspection routines like `fieldnames`

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [November 24, 2020, 2:25pm UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/6 "2020-11-24T14:25:19Z")

</div>

How does it compare to [`Preferences.jl`](https://github.com/JuliaPackaging/Preferences.jl)?

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [November 24, 2020, 2:26pm UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/7 "2020-11-24T14:26:54Z")

</div>

Good point! This is because you still want to distinguish “normal” struct and “option” struct so that we can map hierarchical dictionary better e.g

The from\_dict function is recursively called if there’s no such distinguish then it will convert the whole dictionary to some struct or convert the whole struct recursively to a dictionary which might not be what we want.

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [November 24, 2020, 2:36pm UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/8 "2020-11-24T14:36:15Z")

</div>

I think in general they aim for different things - Pereferences.jl aim for packages so it’s coupled with packages closely while Configurations is used for more general case. You can use it for packages, small scripts, etc and choose to use your own toml or other format or not or use it from CLI inputs

This is why we originally write these things in Pluto and Comonicon- we need a unified representation for options for CLI inputs, keywords argument configuration and maybe a TOML file and it’s not coupled with a certain package but some other project or script user wrote. And propagate this configuration struct to many different functions.

Actually I think you might also end up defining some of your own structs after you get the configuration from Preferences - it’s always convenient to pass only one argument that contains your configuration.

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [November 24, 2020, 2:40pm UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/9 "2020-11-24T14:40:58Z")

</div>

To be short: if you want to config a package use Preference if you as a developer want to let your user configure some of your functionality using either TOML kwargs or other format (e.g CLI inputs) use Configurations.jl

But I do feel this is something should be in stdlib since Pkg created similar structs to handle Project.toml internally as well. Maybe it could be in Preference?

---

<div class="post-metadata">

**Author:** ![benninkrs](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/benninkrs/32/3191_2.png) [@benninkrs](https://discourse.julialang.org/u/benninkrs)\
**Post date:** [November 26, 2020, 3:18am UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/10 "2020-11-26T03:18:06Z")

</div>

Couldn’t you also achieve that behavior by defining an `Option` abstract type to be a common supertype for those structs you want treated as options?

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [November 26, 2020, 3:45am UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/11 "2020-11-26T03:45:57Z")

</div>

yes you can, this is back to the question of whether one should use traits or abstract types. In this case, I think traits are preferred since these options type can be used to dispatch other methods once it finishes reading configurations from kwargs or TOML files. Traits will let you be able to use your own abstract type.

When I implement these option types by hand, yes, I usually create an abstract type, but note this is a more general package to free people from doing this type of work, e.g you have some optimizers that are either gradient based `ADAM` or non-gradient based `CMAES` they may need to be dispatched to different interface since gradient based methods may need AD and non-gradient based methods may need other things.

You would want to declare `ADAM <: GradientMethod` and `CMAES <: NonGradientMethod` rather than make them both subtype of `AbstractConfiguration` - the property of configurable is not an abstract type, but a property that a lot things have, so I choose to not restrict this.

---

<div class="post-metadata">

**Author:** ![moeddel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/moeddel/32/18641_2.png) [@moeddel](https://discourse.julialang.org/u/moeddel)\
**Post date:** [November 26, 2020, 9:12am UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/12 "2020-11-26T09:12:37Z")

</div>

I am currently developing a package,

[https://github.com/MagneticParticleImaging/MPISimulations.jl](https://github.com/MagneticParticleImaging/MPISimulations.jl)

where I face a similar situation. I want to be able to parse structs from TOML files [here](https://github.com/MagneticParticleImaging/MPISimulations.jl/blob/master/src/TOML.jl). My biggest issue is, that the types in my structs are often parametric. E.g.

```julia
struct SinusoidalField{T,U:<StaticMagneticFields{T}}
    spatialFieldProfile::U
    frequency::T
end

```

has the field `spatialFieldProfile` which can be any subtype of the abstract type `StaticMagneticFields{T}` (e.g. `StaticHomogeneousField{T}`). So my approach has been to encode this information into the TOML file like so, which works

```julia
[SinusoidalField]
frequency = 2.5e6
[SinusoidalField.spatialFieldProfile.StaticHomogeneousField]
magneticFieldStrength = [0.12,0.12,0.12]

```

Can you think of a better solution?

A different issue I have are structs which have array and tuple like fields. How to handle the case, where e.g. I want to parse a struct like this on?

```julia
struct foo{N}
    bar::NTuple{N,ComplexF64}
end

```

---

<div class="post-metadata">

**Author:** ![CodeLenz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/codelenz/32/3419_2.png) [@CodeLenz](https://discourse.julialang.org/u/CodeLenz)\
**Post date:** [November 26, 2020, 11:15am UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/13 "2020-11-26T11:15:20Z")

</div>

Hi.

Thanks for this package.

I must be doing something wrong, but

> str = “”"  
> name = “Testing”
> 
> [Optimization]  
> gamma\_1 = 0.95  
> nouter = 10  
> ninner = 2000
> 
> “”"
> 
> fd = open(“config.toml”,“w”)  
> write(fd,str)  
> close(fd)

than\>

> @option struct Optimization  
> gamma\_1::Float64 = 1.0  
> nouter::Int64 = 1  
> ninner::Int64 = 5000  
> end
> 
> @option struct Config
> 
> ```
> opt::Optimization=Optimization()
> 
> name::String=""
> 
> ```
> 
> end
> 
> configuracoes = from\_toml(Config,“config.toml”)

returns

> Config(;  
> opt = Optimization(;  
> gamma\_1 = 1.0,  
> nouter = 1,  
> ninner = 5000,  
> ),  
> name = “Testing”,  
> )

where the field “name” is correct, but the fields under Optimization are still showing the default values.

If I use

> TOML.parsefile(“config.toml”)  
> Dict{String,Any} with 2 entries:  
> “name” =\> “Testing”  
> “Optimization” =\> Dict{String,Any}(“gamma\_1”=\>0.95,“ninner”=\>2000,“nouter”=\>1…

it recovers the correct values from the config file.

What am I missing? Thanks!

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [November 26, 2020, 3:50pm UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/14 "2020-11-26T15:50:00Z")

</div>

It should be your field name rather than your type name, we won’t be able to distinguish different fields with the same type if we use type name do we? The corresponding TOML file should be

```nohighlight
name = "Testing"

[opt]
gamma_1 = 0.95
nouter = 10
ninner = 2000

```

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [November 26, 2020, 4:01pm UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/15 "2020-11-26T16:01:24Z")

</div>

type parameters are things you would like to know before creating this type, thus first it should appear somewhere in your TOML file or they should be inferred from your input type info.

there are cases when you have multiple similar `option` types for one field, e.g in a TOML file

```toml
[MyStruct.field.A]
propertyA = "A"
propertyB = "B"

```

will assign `field` with option type `A` and similarly you can have option type `B` for the same field, in this case, you will need to overload `from_dict` function for `MyStruct` (for current Configurations master branch) then use `default_from_dict` to convert the rest, then you will be able to use multiple types with your own syntax, e.g for this case

```julia
function Configurations.from_dict(::Type{<:MyStruct}, d::AbstractDict{String})
      type = first(keys(d["field"]))
      if type == "A"
          T = AOptions
      elseif type == "B"
          T = BOptions
      end
      MyStruct(default_from_dict(T, d["field"][type]), <other fields>)
end

```

> A different issue I have are structs which have array and tuple like fields. How to handle the case, where e.g. I want to parse a struct like this on?

`Tuple`s are not valid TOML fields, you should either not use them if you want to use TOML file or define your custom conversion rules for it.

* * *

for the multiple option type case I’m think about extending the macro syntax to free people from writing it manually. But it’s not there yet.

---

<div class="post-metadata">

**Author:** ![CodeLenz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/codelenz/32/3419_2.png) [@CodeLenz](https://discourse.julialang.org/u/CodeLenz)\
**Post date:** [November 26, 2020, 4:40pm UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/16 "2020-11-26T16:40:43Z")

</div>

I see. Thanks.

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [November 28, 2020, 9:00pm UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/17 "2020-11-28T21:00:16Z")

</div>

Some new features in the v0.3 version:

## Option Type Alias

now you can declare an alias for your option type, e.g

```julia
@option "option_a" struct OptionA
   field::String = "some field"
end

@option "option_b" struct OptionB
   name::String = "some name"
end

@option struct MyOption
   other::Union{OptionA, OptionB}
end

```

and now `from_toml` function (or similarly fro `from_dict`) can create new instance of such structure by reading the following TOML file

```toml
[other.option_a]
field = "a field"

```

```julia
julia> from_toml(MyOption, "demo.toml")
MyOption(;
    other = OptionA(;
        field = "a field",
    ),
)

```

or it creates `OptionB` for

```toml
[other.option_b]
name = "a name"

```

```julia
julia> from_toml(MyOption, "demo.toml")
MyOption(;
    other = OptionB(;
        name = "a name",
    ),
)

```

## Custom Type Conversion

Different formats support different built-in types and they could be uncompatible with Julia types. Now you can define this by overloading `Configurations.option_convert`, e.g

we can define how to convert `String` to `VersionNumber`

```julia
Configurations.option_convert(::Type{OptionB}, ::Type{VersionNumber}, x::String) = VersionNumber(x)

```

## Default Value Reflection

it is sometimes very useful to be able to access the default value of each field, which `Base.@kwdef` does not support, but now `@option` supports it, one can access the default values via `Configurations.field_defaults(::Type)` as long as you defined your type using `@option`.

* * *

Some of the above features are only possible with `macro`s, thus again, for whoever thought this is easier without macros, you may be able to implement what you need in a specific context, but in the more generic context, the macro `@option` is necessary. But I’m sure some of the efforts in this package could be merged into `Base` or `Preferences` since there are overlapping with `Base.@kwdef` and `Preferences` as well as some of the stdlibs.

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [February 16, 2021, 6:28pm UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/19 "2021-02-16T18:28:23Z")

</div>

Configurations v0.11

new documentation site available here:

[https://rogerluo.me/Configurations.jl/dev/](https://rogerluo.me/Configurations.jl/dev/)

### Option Field Name Alias

[https://rogerluo.me/Configurations.jl/dev/advance/#Field-Name-Alias](https://rogerluo.me/Configurations.jl/dev/advance/#Field-Name-Alias)

Sometimes, the `struct` field name is defined using a UTF-8 character such as `Ω` . It may cause incompatibility when converting to other format such as JSON/TOML or just a `Dict` for JSON-based REST API.

You can define an alias for the field in this case using the following syntax

```julia
@option struct MyType
    "alpha"
    α::Int
    "omega"
    Ω::Float64 = 1.0
end

```

### Custom Option Macro

In some cases, you may not want all the features we defined by default in `Configurations` , such as the printing, etc.

In this case, you can construct your own macro using the code generation passes defined in `Configurations` . The code generation passes API starts with `codegen_` .

`Configurations` uses an intermediate representation to represent user-defined option types, which is the `OptionDef` struct.

e.g you can modify the passes included easily:

```julia
function codegen(def::OptionDef)
    quote
        $(codegen_struct_def(def))
        Core.@ __doc__ $(def.name)
        $(codegen_kw_fn(def))
        $(codegen_convert(def))
        $(codegen_show_text(def))
        $(codegen_is_option(def))
        $(codegen_field_default(def))
        $(codegen_field_alias(def))
        $(codegen_alias(def))
        $(codegen_isequal(def))
        $(codegen_show_toml_mime(def))
        $(codegen_create(def))
    end
end

function option_m(@nospecialize(ex), alias=nothing)
    def = OptionDef(ex, alias)

    quote
        $(codegen(def))
        nothing
    end
end

```

---

<div class="post-metadata">

**Author:** ![JosephPollacco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josephpollacco/32/51404_2.png) [@JosephPollacco](https://discourse.julialang.org/u/JosephPollacco)\
**Post date:** [July 6, 2021, 5:37am UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/20 "2021-07-06T05:37:42Z")

</div>

This is indeed a great software and thanks for developing it, it saved me many lines. I successfully used it to parse_Bool_, _Float_ and _Int_.

**\> But I was not successful to read Symbol into toml file, how do I do ?**

```julia
@option struct DATA
      Kθ::Bool
      Pedological⍰::Symbol
   end # struct DATA

```

TOML FILES

```julia
[data]
    "Pedological⍰" = ":Core" # <:Core>; from traditional data set <:Smap> from Smap; <No> no data available

```

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [July 6, 2021, 6:12am UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/21 "2021-07-06T06:12:02Z")

</div>

Symbol is not a TOML native type, thus you will need to define a type conversion specific to your option type defined above to convert the String to your target object, e.g VersionNumber is supported by default

> <https://github.com/Roger-luo/Configurations.jl/blob/master/src/convert.jl>

---

<div class="post-metadata">

**Author:** ![JosephPollacco](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/josephpollacco/32/51404_2.png) [@JosephPollacco](https://discourse.julialang.org/u/JosephPollacco)\
**Post date:** [July 6, 2021, 6:32am UTC](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669/22 "2021-07-06T06:32:45Z")

</div>

Thanks, I am wandering if YAML format would be a better option as it seems to support symbols and Greek.

> It will be great if you can help to fix **from\_dict** which does not work.

```julia
# DEFINING STRUCTURE
using Configurations, YAML
@option mutable struct EVAPOTRANSPIRATION
      α::Symbol
      Evaporation::Int64
   end

   @option mutable struct SOIL
      Topsoil::Float64 
      Macropore::String 
   end

   @option mutable struct OPTION
      evapotranspiration::EVAPOTRANSPIRATION
      soil::SOIL 
   end

function TOML_TEST()
   # PARSING TOML FILE
      Path= "D:\\Main\\MODELS\\SoilWater-ToolBox2\\src\\Temporary\\Toml.yaml"
      # TomlParse = TOML.parsefile(Path)

      YamlDict = YAML.load_file(Path)

      println(YamlDict)
# Dict{Any, Any}("evapotranspiration" => Dict{Any, Any}("Evaporation" => 1, "α" => ":Large"), "soil" => Dict{Any, Any}("Topsoil" => "Top", "Macropore" => 3))

      option = Configurations.from_dict(OPTION, YamlDict)
      # option = Configurations.from_toml(OPTION, Path)
  
   # TESTING
      println(option.evapotranspiration.Evaporation)
      println(option.evapotranspiration.α)
      println(option.soil.Macropore)
      println(option.soil.Topsoil)
end

```

The input file

```julia
# YAML INPUT FILE
evapotranspiration:
   α: :Large
   Evaporation: 1

soil:
   Macropore: 3
   Topsoil: "Top"

```

**ERROR MESSAGE**

````julia
ERROR: LoadError: MethodError: no method matching from_dict(::Type{OPTION}, ::Dict{Any, Any})
Closest candidates are:
  from_dict(::Type{T}, ::AbstractDict{String, V} where V; kw...) where T at C:\Users\pollaccoj\.julia\packages\Configurations\kq1sK\src\parse.jl:27```
````

[Next page](https://discourse.julialang.org/t/ann-configurations-jl-options-configurations-made-easy/50669.md?page=2)