Is it possible to use environmental variables in YAML.jl?

General Usage
, ,
1

I would like to store relative paths in my YAML file.
Googling, I saw that the way to do it in other languages is to use environmental variables.

So I did try with:

scenario_path = "/some/path/"
settings_path = joinpath(scenario_path,"settings.yaml")
ENV["SCENARIO_PATH"] = scenario_path
settings = YAML.load_file(settings_path)

and, inside the yaml file, with:

ft_list: ${SCENARIO_PATH}/forest_types.txt

However, the ENV variable is not expanded in the output returned by YAML.jl.

Is there a way I can parse environmental variables in YAML.jl, or use another way to have relative paths that expand to an absolute path in a YAML configuration file ?

Of course, I could simply use replace(def_settings["ft_list"],"\${SCENARIO_PATH}" => "/some/path") but this obliges me to keep a list of the places where I use this relative path outside the configuration file…

2

Perhaps not the most elegant solution, but manual replace seems to work:

recursive_replace!(obj,replacement_pattern) = obj
recursive_replace!(obj::AbstractArray,replacement_pattern)  = recursive_replace!.(obj::AbstractArray,Ref(replacement_pattern))
recursive_replace!(obj::AbstractString,replacement_pattern) = replace(obj,replacement_pattern)
function recursive_replace!(obj::AbstractDict,replacement_pattern)
    for (k,v) in obj
        obj[k] = recursive_replace!(v,replacement_pattern)
    end
    return obj
end

function load_settings(project="default",scenario="default")
    rep_path      = joinpath(@__DIR__,"..","repository")
    project_path  = joinpath(rep_path,project)
    scenario_path = joinpath(project_path,"scenarios",scenario)
    settings_path = joinpath(scenario_path,"settings.yaml")
    ENV["SCENARIO_PATH"] = scenario_path
    settings      = YAML.load_file(settings_path)
    recursive_replace!(settings,"\${SCENARIO_PATH}" => scenario_path)
    return settings
end
3

Please note that variable interpolation is not as trivial as it looks, there are many quirks and edge cases. There are also several variants out there. The docker-compose manual has a nice summary of a variation that’s not unlike what bash and zsh do.

Recently, I have implemented my own Julia code for this type of interpolation (with default values, errors, etc.). It’s probably not 100%, but covers most cases.

const UNBRACED_RE = r"\$([A-Z0-9_]+)"
const BRACED_RE   = r"\${([A-Z0-9_]+)(?:(:?)([?+-])((?:[^${}]|(?R))*))?}"

"""
Perform variable interpolation in `str`ing, using variables from `env`.

https://docs.docker.com/compose/environment-variables/env-file/#interpolation
"""
function interpolate(str::AbstractString;
                     env::AbstractDict{<: AbstractString} = ENV
                    )::String
    function unbraced(s::AbstractString)
        (name,) = match(UNBRACED_RE, s)
        return get(env, name, "")
    end
    function braced(s::AbstractString)
        (name, nonempty, mode, param) = match(BRACED_RE, s)
        param = param === nothing ? nothing : interpolate(param; env)
        v = get(env, name, nothing)
        if v !== nothing && (nonempty != ":" || !isempty(v))
            return mode == "+" ? param : v
        else
            return mode == "-" ? param : mode == "?" ? error(param) : ""
        end
    end
    return replace(str, UNBRACED_RE => unbraced, BRACED_RE => braced)
end
4

I’m not aware of any standard YAML reader that does environment variable substitution out of the box, although it’s a common thing to do in various applications that use YAML configuration files. As far as I know it’s done in one of three ways:

  1. Substitution in the data before feeding it to the YAML reader.
  2. Substitution in the output from the YAML reader.
  3. Calling the YAML reader with a custom constructor.
5

If you have envsubst avaiable (standard linux utility from GNU gettext), then you can use it to easily do point (1) in Gunnar’s suggested methods:

julia> write("test.yaml", "ft_list: \${SCENARIO_PATH}/forest_types.txt") # write test file
42

julia> ENV["SCENARIO_PATH"]="mypath" # set env
"mypath"

julia> using YAML

julia> load_and_substitute(file, env...) = YAML.load(IOBuffer(read(pipeline(file, addenv(`envsubst`, env...)))))
load_and_substitute (generic function with 1 method)

julia> load_and_substitute("test.yaml")
Dict{Any, Any} with 1 entry:
  "ft_list" => "mypath/forest_types.txt"

julia> load_and_substitute("test.yaml", "SCENARIO_PATH" => "mypath2")
Dict{Any, Any} with 1 entry:
  "ft_list" => "mypath2/forest_types.txt"

Here I used addenv to let you easily customize the environment for testing etc, but you don’t need it.

There is a Gettext_jll but it doesn’t seem to have envsubst as an executable product. Maybe that can be added in Yggdrasil so then you have a portable version that doesn’t depend on the user having envsubst already installed.