Metaprogramming with Structure: Converting Dictionary to Structure

Dear Julian colleagues,

I am reading the options of my model from TOML file into Structure. With the following steps:

  • The TOML writes into a Dict:
  • Than I am writing a code to parse Dict into Structure

The code works. But the question is how can I remove the if iKey == “evapotranspiration” and elseif iKey == “soil”.
Putting the question in another way is how do I transform a String into Struct. I was not successful in using eval methods.

Many thanks for helping to automatize the process :cowboy_hat_face:
Joseph

# DEFINING STRUCTURE
   mutable struct EVAPOTRANSPIRATION
      Evaporation::Bool
      Transpiration::Bool
   end

   mutable struct SOIL
      Topsoil :: Bool
      Macropore :: Bool
   end

   mutable struct OPTION
      evapotranspiration::EVAPOTRANSPIRATION
      soil::SOIL 
   end

using TOML, StructTypes
function TOML_TEST()
   # PARSING TOML FILE
      PathHome = @__DIR__
      Path = PathHome *  "/Toml.toml"
      # Dict{String, Any}("evapotranspiration" => Dict{String, Any}("Evaporation" => true, "Transpiration" => false), "soil" => Dict{String, Any}("Topsoil" => true, "Macropore" => false))
      TomlParse = TOML.tryparsefile(Path)

   # INITIAL VALUES OF STRUCTURE
      Evaporation::Bool =false
      Transpiration::Bool=false
      evapotranspiration=EVAPOTRANSPIRATION(Evaporation, Transpiration)
   
      Topsoil::Bool=false
      Macropore::Bool=false
      soil = SOIL(Topsoil, Macropore)

   # LOOPING THROUGH THE DICT
   for (iKey, iValue₀) in TomlParse

      for iValue in (keys(iValue₀))
         if iKey == "evapotranspiration"
            setfield!(evapotranspiration, Symbol(iValue), TomlParse[iKey][iValue])

         elseif iKey == "soil"
            setfield!(soil, Symbol(iValue), TomlParse[iKey][iValue])
         end 
      end
   end

   option = OPTION(evapotranspiration, soil)
   
 # OUTPUT IS AS EXPECTED
   println(option.evapotranspiration.Evaporation)
   println(option.evapotranspiration.Transpiration)
   println(option.soil.Topsoil)
   println(option.soil.Macropore)
end
# TOML INPUT FILE
[evapotranspiration]
	Evaporation = true
   Transpiration = false

[soil]
  Topsoil = true
	Macropore = false
3 Likes

@JosephPollacco
A few caveats… Not an expert in Julia. I’m very much a beginner. Also, I don’t know your end goal or the edge cases but here is an option without the loop at all:

TomlParse = TOML.tryparse(strs)

splat1 = NamedTuple{Tuple(Symbol.(keys(TomlParse["evapotranspiration"])))}(values(TomlParse["evapotranspiration"]))
splat2 = NamedTuple{Tuple(Symbol.(keys(TomlParse["soil"])))}(values(TomlParse["soil"]))

output = OPTION(EVAPOTRANSPIRATION(splat1...), SOIL(splat2...))

There may be another way without having to convert to a named tuple but nothing I tried worked so far.

If I think of anything else I will post.

2 Likes

Dear @stepenl you should be a genius :cowboy_hat_face: if you are a beginner. You solved my problem beautifully:

  1. The Toml file does not need to be ordered as in the structure;
  2. You do not need to predeclare the STRUCTURE e.g.
      Evaporation::Bool =false
      Transpiration::Bool=false
      evapotranspiration=EVAPOTRANSPIRATION(Evaporation, Transpiration)

Below is the solution of your code

# DEFINING STRUCTURE
   mutable struct EVAPOTRANSPIRATION
      Evaporation::Bool
      Transpiration::Bool
   end

   mutable struct SOIL
      Topsoil :: Bool
      Macropore :: Bool
   end

   mutable struct OPTION
      evapotranspiration::EVAPOTRANSPIRATION
      soil::SOIL 
   end

using TOML
function TOML_TEST()

   # PARSING TOML FILE
      Path= "D:\\Main\\MODELS\\SoilWater-ToolBox2\\src\\Temporary\\Toml.toml"
      TomlParse = TOML.parsefile(Path)

   # STRUCTURE
      evapotranspiration = TOML_2_STRUCT(EVAPOTRANSPIRATION, TomlParse; MyType_LowerCase=true)

      soil = TOML_2_STRUCT(SOIL,TomlParse; MyType_LowerCase=true)

   option = OPTION(evapotranspiration, soil)
   
   # TESTING
      println(option.evapotranspiration.Evaporation)
      println(option.evapotranspiration.Transpiration)
      println(option.soil.Topsoil)
      println(option.soil.Macropore)
end


   function TOML_2_STRUCT(Structure, TomlParse; MyType_LowerCase=true, MyType=:MyType)
      if MyType_LowerCase == false
         MyType = string(MyType)
      else
         MyType = lowercase.(string(Structure))
      end

      Output = NamedTuple{Tuple(Symbol.(keys(TomlParse[MyType])))}(values(TomlParse[ MyType]))
     return Structure(Output...)
   end

I do something similar and what helped me was:

  • You could use Base.@kwdef to give the structs default values directly.
  • There is a package Home · StructTypes.jl which is well suited for the task of converting a Dict into structs.

Anyway, since you have already a working solution, this might not be needed anymore :slight_smile:

Hi Stephen,

It will be informative if you could give an example how to use Home · StructTypes.jl as the documentation is a bit overwhelming for simple users.

I think this section might be most helpful

https://juliadata.github.io/StructTypes.jl/stable/#StructTypes.AbstractType

1 Like

Not an answer to your question, but have you seen [ANN] Configurations.jl - Options/Configurations made easy? May be there is no need to write your own implementation.

2 Likes

Yeah basically just replace the kwdef macro with option in Configuration then it should just work.

@JosephPollacco A warning, that code it’s not doing what you think it is doing.

In the the call EVAPOTRANSPIRATION(splat1...), the values of the named tuple are given to EVAPOTRANSPIRATION in the order of the construction of the named tuple (the keys are simply ignored). In other words, if the keys are reversed in the TOML file, the values of the struct will be reversed, too.

Using Base.@kwdef and (; kwargs...) should solve that issue, and kwargs can be a NamedTuple or any dictionary with Symbol keys. The semicolon is needed for Julia to interpret the keys and values in kwargs as keyword arguments.

@kwdef mutable struct EVAPOTRANSPIRATION
    Evaporation::Bool
    Transpiration::Bool
end

@kwdef mutable struct SOIL
    Topsoil :: Bool
    Macropore :: Bool
end

@kwdef mutable struct OPTION
    evapotranspiration::EVAPOTRANSPIRATION
    soil::SOIL 
end

using TOML

TomlParse = TOML.tryparse(strs)

splat1 = NamedTuple{Tuple(Symbol.(keys(TomlParse["evapotranspiration"])))}(values(TomlParse["evapotranspiration"]))
splat2 = NamedTuple{Tuple(Symbol.(keys(TomlParse["soil"])))}(values(TomlParse["soil"]))

evapotrans = EVAPOTRANSPIRATION(; splat1...)
soil = SOIL(; splat2...)

output = OPTION(evapotrans, soil)
2 Likes

Thanks for the correction. When I use code like this I have a constructor for the struct using keyword arguments. I forgot to add that to the example code. I’ve never used the @kwef macro before.

1 Like

Thanks @jessymilare for your response to improve the robustness of the code. I am not sure if I understood the advantages of using Base.@kwdef as in the code below seems to work independently of the order of TOML input:

OPTION 1

[evapotranspiration]
   Transpiration = 2
	Evaporation = 1
[soil]
	Macropore = false
  Topsoil = true

OPTION 2

[evapotranspiration]
   Evaporation = 1
   Transpiration = 2

[soil]
	Macropore = false
  Topsoil = true

The code I am using is:

   mutable struct EVAPOTRANSPIRATION
      Evaporation
      Transpiration
   end
mutable struct SOIL
      Topsoil :: Bool
      Macropore :: Bool
   end
mutable struct OPTION
      evapotranspiration::EVAPOTRANSPIRATION
      soil::SOIL 
   end

using TOML
function TOML_TEST()
   # PARSING TOML FILE
      Path= "D:\\Main\\MODELS\\SoilWater-ToolBox2\\src\\Temporary\\Toml.toml"
      TomlParse = TOML.parsefile(Path)

   # STRUCTURE
      evapotranspiration = TOML_2_STRUCT(EVAPOTRANSPIRATION, TomlParse; MyType_LowerCase=true)

     soil = TOML_2_STRUCT(SOIL,TomlParse; MyType_LowerCase=true)

   option = OPTION(evapotranspiration, soil)
   
   # TESTING
      println(option.evapotranspiration.Evaporation)
      println(option.evapotranspiration.Transpiration)
      println(option.soil.Topsoil)
      println(option.soil.Macropore)
end

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#		FUNCTION : TOML_2_STRUCT
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   function TOML_2_STRUCT2(Structure, TomlParse)
      # LOOPING THROUGH THE DICT
      for (iKey, iValue₀) in TomlParse
      for iValue in (keys(iValue₀))
         if uppercase.(iKey) == (string(typeof(Structure)))
            setfield!(Structure, Symbol(iValue), TomlParse[iKey][iValue])
         end 
      end
   end
   return Structure
   end  # function: TOML_2_STRUCT

   function TOML_2_STRUCT(Structure, TomlParse; MyType_LowerCase=true, MyType=:MyType)
      if MyType_LowerCase == false
         MyType = string(MyType)
      else
         MyType = lowercase.(string(Structure))
      end

      Output = NamedTuple{Tuple(Symbol.(keys(TomlParse[MyType])))}(values(TomlParse[MyType]))
     return Structure(Output...)
   end```

The solution by @Skoffer works remarkably well by using Configurations Configuration.jland is the solution I was looking for as it can read complex TOML structure file automatically.

My recommendation is that Configuration.jl should be part of the TOML package.

using Configurations
@option  mutable struct EVAPOTRANSPIRATION
      Evaporation
      Transpiration
   end

   @option  mutable struct SOIL
      Topsoil 
      Macropore 
   end

   @option  mutable struct OPTION
      evapotranspiration::EVAPOTRANSPIRATION
      soil::SOIL 
   end

function TOML_TEST()
   # PARSING TOML FILE
      Path= "D:\\Main\\MODELS\\SoilWater-ToolBox2\\src\\Temporary\\Toml.toml"
      # TomlParse = TOML.parsefile(Path)

      option = Configurations.from_toml(OPTION, Path)
  
   # TESTING
      println(option.evapotranspiration.Evaporation)
      println(option.evapotranspiration.Transpiration)
      println(option.soil.Macropore)
      println(option.soil.Topsoil)
end

The problem I refered to still exists with you function TOML_2_STRUCT. It seems to be working by coincidence (TOML creates a Dict, which sorts by hash).

Let me try to illustrate the issue.

julia> mutable struct EVAPOTRANSPIRATION
           Evaporation
           Transpiration
       end

julia> EVAPOTRANSPIRATION(true, false)
EVAPOTRANSPIRATION(true, false)

julia> EVAPOTRANSPIRATION(false, true)
EVAPOTRANSPIRATION(false, true)

julia> EVAPOTRANSPIRATION(Evaporation = true, Transpiration = false)
ERROR: MethodError: no method matching EVAPOTRANSPIRATION(; Evaporation=true, Transpiration=false)
Closest candidates are:
  EVAPOTRANSPIRATION(::Any, ::Any) at REPL[1]:2 got unsupported keyword arguments "Evaporation", "Transpiration"
Stacktrace:
 [1] top-level scope
   @ REPL[4]:1

See? The constructor EVAPOTRANSPIRATION accepts positional arguments, not keyword arguments. (Keyword arguments are arguments with names).

This is what is what might eventually happen inside your TOML_2_STRUCT:

julia> Output = (Transpiration = false, Evaporation = true)
(Transpiration = false, Evaporation = true)

julia> EVAPOTRANSPIRATION(Output...)
EVAPOTRANSPIRATION(false, true)

julia> ans.Evaporation
false

The variable Output contains a NamedTuple with :Evaporation key set to true, but the constructed EVAPOTRANSPIRATION has the first field (which is Evaporation) set to false.

What is happening is this. The call EVAPOTRANSPIRATION(Output...) is evaluated as EVAPOTRANSPIRATION(false, true) because the values of the named tuple are spliced as positional arguments and the key names are ignored. So, the false (which refers to :Transpiration) is supplied as first argument for EVAPOTRANSPIRATION, which refers to Evaporation field.

To resolve that, you need to use Base.@kwdef, which creates a constructor that accepts keyword arguments, AND to change EVAPOTRANSPIRATION(Output...) into EVAPOTRANSPIRATION(; Output...) that is, with a semicolon before Output so that it is spliced as keyword arguments instead of positional arguments.

Another example with keyword arguments:

julia> function foo(; x = 0, y = 0)
           x + y
       end
foo (generic function with 1 method)

julia> foo(x = 1, y = 2)
3

julia> kwargs = (x = 10, y = 20)
(x = 10, y = 20)

julia> foo(kwargs...)
ERROR: MethodError: no method matching foo(::Int64, ::Int64)
Stacktrace:
 [1] top-level scope
   @ REPL[21]:1

julia> foo(; kwargs...)
30

foo accepts no positional argument and two keyword arguments. kwargs is a named tupe and foo(kwargs...) doesn’t work because the arguments are supplied as positional arguments. But foo(; kwargs...) works because the semicolon tells Julia that the named tuple kwargs must be spliced as keyword arguments.

3 Likes

@jessymilare a great thanks for providing the community such great explanations which explains beautifully about positional arguments.

1 Like