[ANN] Configs.jl: Opinionated tool for managing deployment configurations

Package Announcements
1

I am new to Julia, so firstly a “Hello” to everybody in the community.

Coming from a Node.js background, I miss some generic tools that I have become accustomed to, specifically managing deployment configurations with the Node config package.

For this purpose, I have made Julia Configs as a clone of the aforementioned Node package.

Methodology:
Configs allows you define and then contextually override deployment configurations using ENVIRONMENT variables.

This works in a cascading priority:

  1. JSON file default configuration
  2. JSON file deployment specific configuration [development, staging, production, etc]
  3. JSON file mapping of ENVIRONMENT variables to configuration paths
  4. Code setting of configuration value

After the first code retrieval of a configuration, Configs becomes immutable.

Illustrative usage:

$> export DEPLOYMENT=staging
$> export DATABASE_PASSWORD=supersecret
$> julia --project=. src/myproject.jl

using Configs
externalconfigvalue = getfromexternal(...) # retrieve configurations from external sources

setconfig!("my.external.config", externalconfigvalue)
hasconfig("my.external.config") #true
#config has been retrieved, so Configs are now immutable

database = getconfig("connections.database") #NamedTuple
url = database.url
port = database.port
username = database.username
password = database.password #"supersecret" was passed in through ENV["DATABASE_PASSWORD"]

This is my first trip into Julialand, so suggestions and criticism on the package will be helpful.

5 Likes
2

Nice! I have some similar functionality in https://github.com/oschulz/PropDicts.jl (merging/cascading Dicts), but no direct assignment of “my.external.config”, if that property doesn’t exist yet. However, I do support setproperty! for JS-like unquoted property access. I wonder if that might be possible in Configs.jl too, creating temporary nodes on the fly if necessary?

1 Like
3

I want to preserve immutability after all the setconf! calls are done, so an “on the fly” temporary node is unwanted. This is also how Node conf works, and the principle is that configurations should never be able to cause a side affect once set.

setconfig! is pretty flexible:

setconfig!("database.connection.port", 2000)
setconfig!("database", Dict("connection" => Dict("port" => 2000)))

You have made me think that this could also be useful:

setconfig!("database", """{
    "connection": {
        "port": 2000
    }
}""")

so that retrieving a bulk JSON payload from external source can be input directly.

4

This was on my wishlist, thanks for creating it!

I would suggest adding a memoization layer, as the performance is not very good (9-25x slower than a Dict{String, Any} in my simple test).

5

Thanks, I will do some research into that as I have not used Memoization before.
Would you suggest https://github.com/JuliaCollections/Memoize.jl?

6

While writing this post I just found that calling getconfig() without arguments gives you a recursive NamedTuple with the full config, which is best for performance! So please let me change my original suggestion to only documenting this a bit. :slight_smile:

If still interested in memoization:

Memoize.jl is definitely a way to go, just tried it out:

  | | |_| | | | (_| |  |  Version 1.5.1 (2020-08-25)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Configs, Memoize, BenchmarkTools

julia> setconfig!("database.connection.port", 42)
42

julia> @btime getconfig("database.connection.port")
  462.756 ns (12 allocations: 400 bytes)
42

julia> @memoize Dict{Tuple{String}, Any} function memoized_getconfig(key)
           return getconfig(key)
       end
memoized_getconfig (generic function with 1 method)

julia> @btime memoized_getconfig("database.connection.port")
  30.546 ns (0 allocations: 0 bytes)
42

This is not bad, and better configuration may be possible, Dict{Tuple{String}, Any} was my best.

I think that manually filling a Dict{String, Any} would get you under 20ns without the extra dependency, and I feel that no better solution exists for String keys.

7

Ah, right - sorry, I’m not really familiar with Node conf …

8

I have spent the day experimenting with Julia data structures and memoize.

What Configs is doing in the background, after the first call to hasconfig or getconfig, is to make the configs data immutable by converting from Dict{String, Any} to a nested tree of NamedTuples and Tuples.

This ensures that the config is immutable, and provides the my.config.path inline syntax, but this appears to put the performance issue into a Mutual Recursion domain.

For this reason, I have implemented memoize on the recursion, which has improved general performance.

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.0 (2020-08-01)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Configs, BenchmarkTools

julia> @btime getconfig("database.connection.port")
  29.478 ns (0 allocations: 0 bytes)
3600

julia> @btime database = getconfig("database")
  26.445 ns (0 allocations: 0 bytes)
(empty = NamedTuple(), credentials = (password = "guestuserdefault", username = "guest"), connection = (url = "http://localhost", port = 3600))

julia> database = getconfig("database")

julia> @btime database.connection.port
  60.754 ns (4 allocations: 128 bytes)
3600

julia> connection = getconfig("database.connection")
(url = "http://localhost", port = 3600)

julia> @btime connection.port
  29.742 ns (2 allocations: 48 bytes)
3600

It is interesting when accessing the NamedTuple through database.connection.port progressively degrades performance significantly with every branch extension. This is overcome with memoize, so best performance will be by using getconf as close to your config logic cluster, then drilling down further using . syntax.

Anyway - I have released v0.2.1 to JuliaHub, and it should be available shortly.

All interesting stuff and I am sure there must be better ways of going, but this will do for now. Thanks for all the pointers.

9

This is a tricky detail of benchmarking and global variables: Global variables are slow, because the compiler cannot be sure that the type of the value in the variable will never change, and it generates code for handling that. So when measuring performance from the REPL, you should use constants or interpolate with $.

  | | |_| | | | (_| |  |  Version 1.5.1 (2020-08-25)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Configs, BenchmarkTools

julia> const dbconf = getconfig("database")
(credentials = (password = "guestuserdefault", username = "guest"), connection = (url = "http://localhost", port = 3600))

julia> @btime dbconf.connection.port
  0.015 ns (0 allocations: 0 bytes)
3600

julia> vardbconf = getconfig("database")
(credentials = (password = "guestuserdefault", username = "guest"), connection = (url = "http://localhost", port = 3600))

julia> @btime $vardbconf.connection.port
  0.015 ns (0 allocations: 0 bytes)
3600

0.015ns is too low to be true, it means that the value was constant propagated.

Did you think of supporting alternative formats? I think without evidence that YAML is more popular than JSON for storing configuration data. Anyway, I find it nicer for configuration, and JSON has no “native syntax” advantage outside the JS world.

1 Like
10

Yes - I am thinking YAML and also plain .jl DICT for every config. Any thoughts on TOML?

I am still very much coming from a JS frame of mind, and one of the reasons I am building this package is to help me migrate to a Julian way before I launch into the project I really want to build.

Thanks for the hints about constants and $ interpolation. I will do some research and wrap that around my head.

11

No thoughts on TOML vs YAML, but reading Julia code as config would be very nice, I was thinking of this before, and although it is a bit scary maintainability-wise, I would definitely give it a try.

I would prefer using NamedTuples for the simpler syntax, e.g. something like this in configs/default.jl:

module DefaultConfig

getconfig() = (
    otherstuff = (
        defaultmessage = "Hello $(rand())"
    ),
    database = ( 
        connection = (
            url = "http://localhost",
            port = 42
        ),
        credentials = (
            username = "guest",
            password = "guestuserdefault"
        )
    )
)

end

edit: The need of trailing commas when only one item is in the NamedTuple makes this format a bit error-prone, as seen from my mistake in the sample (at defaultmessage).

1 Like
12

For data like this, it is good to get into the habit of having trailing commas at the end of each line, as the last one is innocuous and it makes a lot of things easier (rearranging, change tracking in version control, etc):

julia> (a = 1,
        b = 2,
       )
(a = 1, b = 2)
3 Likes
13

Configs should bump to v0.2.2 on Juliahub any minute now.

This version allows for defining configs in multiple file formats:

  • .jl
  • .yml
  • .json

@Tamas_Papp Regarding .jl NamedTuple formatting, it will be down to individual user’s housekeeping to use trailing ,'s in config files. Configs is agnostic, and will accept any standard collection type:

  • Tuples
  • NamedTuples
  • Array
  • Dict

or combinations of these. More in the updated docs.

If anybody has time for such a thing, a critical code review will be helpful. (233 lines including comments)
This is my first Julia project, so more experienced eyes will be wise before this package is used for production.

2 Likes