# Hand-editable serialization: JSON, Fortran's namelist, etc

**URL:** <https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468>\
**Category:** General Usage\
**Created:** [June 16, 2023, 7:09pm UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468 "2023-06-16T19:09:23Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![ryofurue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ryofurue/32/24531_2.png) [@ryofurue](https://discourse.julialang.org/u/ryofurue)\
**Post date:** [June 16, 2023, 7:09pm UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/1 "2023-06-16T19:09:23Z")

</div>

I don’t know the issue well enough to be able to come up with an apt title to this thread.

My question is, what are the common, hand-editable data formats to initialize variables?

Suppose you want to supply various parameters and values to your Julia program from a hand-editable text file. What format would you use?

When I was using Fortran, the namelist was the obvious and most convenient format, because you can specify the names of the variables and their values in the text file. Instead of showing the exact format of Fortran’s namelist, I show a pseudo code:

```fortran
# namelistfile.txt
arr = 1,3,5,9
s = "hellow world"
c = 3.0 + 2im
# pseudo Fortran code
integer:: a[4]
string:: s
complex:: c
namelist/myparameters/ arr, s, c
filehandle = open("namelistfile.txt")
read(filehandle, namelist=myparameters) # -> arr, s, and c are initialized

```

What do julia programmers use in such a case as this? You don’t want to invent an ad-hoc data format and write an ad-hoc parser. Perhaps you use JSON?

In my particular applications, I need to be able to express repetition:

```fortran
a = 1, 2, 5*3, 4

```

instead of

```fortran
a = 1,2,3,3,3,3,3,4

```

If there is no such a convenient format, I would perhaps just write a julia module that contains the `const`ants and use it as if it were a datafile and load it via:

```julia
include(ARGS[1])
using .Parameters

```

A downside of this approach (shared by Fortran’s namelist) is that it’s very hard to use the datafile from other languages. If that’s important enough, perhaps I should use some common format like JSON. . . .

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [June 16, 2023, 7:27pm UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/2 "2023-06-16T19:27:45Z")

</div>

I usually prefer to just use Julia source as the serialization format, if Julia is going the be the only consumer.

Note (perhaps this is tidier than the more obvious approach of `include`ing an entire script or module), it’s possible to `include` a Julia expression:

expr.jl:

```julia
vcat(1, 2, [3 for i ∈ 1:5], 4)

```

main.jl:

```julia
const a = (include("expr.jl"))

```

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [June 16, 2023, 7:32pm UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/3 "2023-06-16T19:32:06Z")

</div>

> [@ryofurue](#):
>
> If there is no such a convenient format, I would perhaps just write a julia module that contains the `const`ants and use it as if it were a datafile and load it via:
> 
> ```julia
> include(ARGS[1])
> using .Parameters
> 
> ```

Small note: I think the `using .Parameters` is redundant, although it might be necessary in other cases (for example if the module hierarchy is more complex so you have to do something like `using ..Parameters`).

---

<div class="post-metadata">

**Author:** ![martin.d.maas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/martin.d.maas/32/50964_2.png) [@martin.d.maas](https://discourse.julialang.org/u/martin.d.maas)\
**Post date:** [June 17, 2023, 12:14am UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/4 "2023-06-17T00:14:31Z")

</div>

> [@ryofurue](#):
>
> You don’t want to invent an ad-hoc data format and write an ad-hoc parser.

I know writing a parser could sound intimidating at first, specially coming from languages like Fortran or C++, but it is actually very easy to do in Julia.

I’m right now working on some parsing, and I found out that, for simple cases, all you need are a few string functions (like split), maybe store stuff in a `Dict` with variable names and values, and eventually just run `eval`.

> [@ryofurue](#):
>
> In my particular applications, I need to be able to express repetition:
> 
> ```julia
> a = 1, 2, 5*3, 4
> 
> ```
> 
> instead of
> 
> ```julia
> a = 1,2,3,3,3,3,3,4
> 
> ```

I’m not aware that this is currently implemented in any present data format, so perhaps you do need to parse your text file with a custom function, which could be just a few lines of Julia.

For example, I would start with something like this.

So let’s say you read the first line in your file and get

```julia
str = "a = 1, 2, 5*3, 4"

```

You can then do:

```julia
var, exp = strip.(split(str,"="))
L = strip.(split(exp,","))

```

and then loop over the elements of L to find wether `occursin("*",element)` is true, and process that element.

---

<div class="post-metadata">

**Author:** ![favba](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/favba/32/2735_2.png) [@favba](https://discourse.julialang.org/u/favba)\
**Post date:** [June 17, 2023, 2:01pm UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/5 "2023-06-17T14:01:30Z")

</div>

One straightforward option for config files in Julia is a TOML file, which is used by Julia itself for Projects and Manifests.  
A TOML file reader is part of the [standard library](https://docs.julialang.org/en/v1/stdlib/TOML/).

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [June 17, 2023, 2:19pm UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/6 "2023-06-17T14:19:46Z")

</div>

Well, for files that are hand editable I prefer YAML: [GitHub - JuliaData/YAML.jl: Parse yer YAMLs](https://github.com/JuliaData/YAML.jl)

Update: Both TOML and YAML support comments.

---

<div class="post-metadata">

**Author:** ![favba](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/favba/32/2735_2.png) [@favba](https://discourse.julialang.org/u/favba)\
**Post date:** [June 17, 2023, 4:04pm UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/7 "2023-06-17T16:04:37Z")

</div>

For simple settings, without arrays of deep nested structures or other complex structures, I think TOML is simpler and less error prone (not white-space/tabs dependent).  
But things get really ugly with arrays or complex structures.  
For more complex cases I do think YAML is a better choice.

---

<div class="post-metadata">

**Author:** ![ryofurue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ryofurue/32/24531_2.png) [@ryofurue](https://discourse.julialang.org/u/ryofurue)\
**Post date:** [June 22, 2023, 7:24am UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/8 "2023-06-22T07:24:50Z")

</div>

> I think the `using .Parameters` is redundant,

But I don’t know how to make names in a module available without using `using`:

```julia
# -- contents of samplemodule.jl ---
module Sample
export isample, csample
const isample = 3
const csample = 4 + 5im
end
# -- The main program ---
include("samplemodule.jl")
# using .Sample # -- doesn't work without this line.
println(Sample.csample) # works
println(isample) # fails without `using .Sample`

```

In addition, sometimes I want to import only some of the names:

```julia
using .Sample: isample

```

---

<div class="post-metadata">

**Author:** ![ryofurue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ryofurue/32/24531_2.png) [@ryofurue](https://discourse.julialang.org/u/ryofurue)\
**Post date:** [June 22, 2023, 8:08am UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/9 "2023-06-22T08:08:48Z")

</div>

> [@martin.d.maas](#):
>
> I know writing a parser could sound intimidating at first, specially coming from languages like Fortran or C++, but it is actually very easy to do in Julia.
> 
> I’m right now working on some parsing, and I found out that, for simple cases, all you need are a few string functions (like split), maybe store stuff in a `Dict` with variable names and values,

Sorry that I wasn’t clear in my initial post. I perfectly know what you say.

It’s not the difficulty of wring a parser for a simple grammar. An ad-hoc grammar is fragile for future changes. When you write the parser, you make a lot of _implicit_ assumptions about the input. Then, in the future, when you extend the format of your input file, you break some of the assumptions you didn’t know you made and your program would sometimes fail until you realize the error and fix the parser.

A friend of mine is a programmer. Her program receives information from various external machines (hardware) in various, very simple, ad-hoc grammars. Sometimes the maker of a machine changes the format of its output, causing her program to fail in a mysterious way. After some debugging she discovers this change and modifies her parser. That’s one of her frequent problems. If hardware makers used one or other well-known formats such as YAML, changes to the data would not silently introduce strange values to her program. Depending on what changes are made to the input data, her program would likely detect the change and issue an appropriate error message.

For this reason, it’s often better to adopt one or another widely-used format, rather than writing a parser for an ad-hoc format. If you need only simple values, you may want to use INI, for example. It doesn’t seem hard to write a parser for INI files, but then there is already a package for that.

---

<div class="post-metadata">

**Author:** ![ryofurue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ryofurue/32/24531_2.png) [@ryofurue](https://discourse.julialang.org/u/ryofurue)\
**Post date:** [June 22, 2023, 8:17am UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/10 "2023-06-22T08:17:00Z")

</div>

Thank you all for your ideas and helps!

For using various formats like TOML, I have a question. How do you turn the values in the file to julia variables? (Using a julia module as an input file doesn’t involve that problem.)

Thinking of that, I’m struck by this comment:

> [@martin.d.maas](#):
>
> and eventually just run `eval`.

I don’t know how `eval` works in Julia, but guessing from other languages, you first build a text string which is a snippet of julia code, and evaluate it, thereby turning values in the text file into julia variables.

Perhaps, julia packages for YAML, TOML, etc. already do that for you?

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [June 22, 2023, 10:02am UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/11 "2023-06-22T10:02:07Z")

</div>

You could make a struct for the parameters which will predefine the variables instead of using `eval`. For example,

```julia
struct params 
   a
   b 
   c
end
const p = params() # create an instance of params (you could also create this instance locally within a function and pass it around as arguments

function init_params() 
   toml_data = read_toml_file() 
   p.a = toml_data["a"] 
   p.b = toml_data["b"] 
   p.c = toml_data["c"]
end

```

note that the above is pseudocode, but this is usually how I do it.

---

<div class="post-metadata">

**Author:** ![martin.d.maas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/martin.d.maas/32/50964_2.png) [@martin.d.maas](https://discourse.julialang.org/u/martin.d.maas)\
**Post date:** [June 22, 2023, 10:18am UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/12 "2023-06-22T10:18:00Z")

</div>

> [@ryofurue](#):
>
> How do you turn the values in the file to julia variables? (Using a julia module as an input file doesn’t involve that problem.)

It depends on whether you know the names or your variables in advance or not. If I remember correctly, in Fortran namespaces you do. You declare a namespace with a list of variable names.

If this is the case, instead of using eval, you can “unpack” your Dict. I’m not in a computer now so I can’t check if the same syntax for unpacking named tuples works, but it is likely.

```julia
(; a, b, c, x, y, z) = Params

```

Where Params is a Dict (which is what you get when parsing Toml for example).

---

<div class="post-metadata">

**Author:** ![martin.d.maas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/martin.d.maas/32/50964_2.png) [@martin.d.maas](https://discourse.julialang.org/u/martin.d.maas)\
**Post date:** [June 22, 2023, 11:45am UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/13 "2023-06-22T11:45:16Z")

</div>

My bad, Dicts cannot be unpacked directly as named tuples, but a Dict can be converted to a Named Tuple with a helper function like this one

```julia
dict2ntuple(d) = NamedTuple{Tuple(Symbol.(keys(d)))}(values(d))

```

and then unpacked

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [June 22, 2023, 1:07pm UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/14 "2023-06-22T13:07:56Z")

</div>

> [@ryofurue](#):
>
> But I don’t know how to make names in a module available without using `using`:

Sorry, I prefer `import` over `using` in my packages, so as not to pollute the global namespace, so it simply hadn’t occured to me that’s what you’re after. Nevermind.

---

<div class="post-metadata">

**Author:** ![ryofurue](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ryofurue/32/24531_2.png) [@ryofurue](https://discourse.julialang.org/u/ryofurue)\
**Post date:** [June 23, 2023, 2:48pm UTC](https://discourse.julialang.org/t/hand-editable-serialization-json-fortrans-namelist-etc/100468/15 "2023-06-23T14:48:17Z")

</div>

> (; a, b, c, x, y, z) = Params

Thank you for your help! I get the idea. I think I could implement that.

But that means that there is no existing library that does all these for you, perhaps by using `eval` internally.

You can always use a julia module as a parameter file:

```julia
include("Pars.jl")
# use Pars.a, Pars.b, etc.

```

Here the `include()` function doesn’t know what variables are in the module `Pars`.

Likewise, writing a TOML file, you would be able to say something like:

```julia
import_vars("Pars.toml")
# use Pars.a, Pars.b, etc.

```

where the function `import_vars()` doesn’t know what variables are in the TOML file. . . .

I think you could write such a function by using `eval` (and perhaps `Meta.parse` ?) . . . but perhaps that would be overkill unless you really use this method often.