# Read data file and create a dictionary from the metadata
**URL:** https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200
**Category:** General Usage
**Tags:** question, dictionary, metadata
**Created:** [April 6, 2023, 10:54pm UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200 "2023-04-06T22:54:14Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@Austin\_Weber](https://discourse.julialang.org/u/Austin_Weber)
#### Post date: [April 6, 2023, 10:54pm UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/1 "2023-04-06T22:54:14Z")
I have a spectral data file in which the contents are stored like this:
```julia
#FORMAT : EMSA
#VERSION : 1.0
#TITLE : 2023-Apr-06
#NPOINTS : 4096
#NCOLUMNS : 1
#SPECTRUM :
0.00, 0.0
10.00, 4.0
20.00, 2.0
30.00, 7.0
40.00, 15.0
50.00, 21.0
.
.
.
```
As you can see, the metadata are stored at the top of the file, and I want to create a dictionary of this metadata without having to manually generate it (like below):
```julia
# Manually-generated dictionary of metadata
meta_dict = Dict("#FORMAT" => "EMSA", "#VERSION" => 1.0, "#TITLE" => "2023-Apr-06", "#NPOINTS" => 4096, "#NCOLUMNS" => 1)
```
In the actual data file there is much more than just 5 lines of metadata, and so I am looking for a way to avoid having to manually generate a dictionary.
---
### Author:  [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)
#### Post date: [April 6, 2023, 11:42pm UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/2 "2023-04-06T23:42:33Z")
Some ideas (edited):
```julia
str = """
#FORMAT : EMSA
#VERSION : 1.0
#TITLE : 2023-Apr-06
#NPOINTS : 4096
#NCOLUMNS : 1
#SPECTRUM :
0.00, 0.0
10.00, 4.0
20.00, 2.0
30.00, 7.0
40.00, 15.0
50.00, 21.0
"""
io = IOBuffer(str)
lines = readlines(io)
ix = findfirst(x->first(x)!='#', lines) - 1
meta = split.(chop.(lines[1:ix], head=1, tail=0),":")
dic = Dict(strip.(first.(meta)) .=> strip.(last.(meta)))
```
---
### Author:  [@Austin\_Weber](https://discourse.julialang.org/u/Austin_Weber)
#### Post date: [April 7, 2023, 12:26am UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/3 "2023-04-07T00:26:22Z")
Your solution works pretty well, thank you. The only issue is that in this case the keys and values in the dictionary maintain the white space before and after the colon. For instance, instead of getting
`"#FORMAT" => "EMSA"`
the output is
`"#FORMAT " => " EMSA"`
But I have worked out a way to fix this using a for-loop!
```julia
new_dic = Dict()
for (key, val) in dic
new_key = strip(key)
new_value = strip(val)
new_dic[new_key] = new_value
end
```
Now the code is working how I want it to. I appreciate the help.
---
### Author:  [@NicholasWMRitchie](https://discourse.julialang.org/u/NicholasWMRitchie)
#### Post date: [April 7, 2023, 1:21am UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/4 "2023-04-07T01:21:55Z")
Austin, You might find the package NeXLSpectrum ([GitHub - usnistgov/NeXLSpectrum.jl: EDS spectrum analysis tools within the NeXL toolset](https://github.com/usnistgov/NeXLSpectrum.jl)) interesting. For one, it reads EMSA spectrum files.
---
### Author:  [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)
#### Post date: [April 7, 2023, 5:33am UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/5 "2023-04-07T05:33:30Z")
I think we just need to strip it here:
```julia
dic = Dict(strip.(first.(meta)) .=> strip.(last.(meta)))
```
I have edited the code above.
---
### Author:  [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)
#### Post date: [April 7, 2023, 7:25am UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/6 "2023-04-07T07:25:15Z")
```julia
data="""#FORMAT : EMSA
#VERSION : 1.0
#TITLE : 2023-Apr-06
#NPOINTS : 4096
#NCOLUMNS : 1
#SPECTRUM :
0.00, 0.0
10.00, 4.0
20.00, 2.0
30.00, 7.0
40.00, 15.0
50.00, 21.0
"""
io=IOBuffer(data)
el=eachline(io)
md=startswith("#")
mdd=Dict{String, Any}()
itr,_=iterate(el)
while md(itr)
k=findfirst(' ',itr)-1
v=findfirst(':',itr)+2
mdd[itr[begin:k]]=itr[v:end]
itr,_=iterate(el)
end
mdd
```
using DIctionaries preserves the order of the metadata
```julia
using Dictionaries
julia> @btime begin
io=IOBuffer(data)
el=eachline(io)
md=startswith("#")
mdd=Dictionary{String, Any}()
itr,_=iterate(el)
while md(itr)
k=findfirst(' ',itr)-1
v=findfirst(':',itr)+2
insert!(mdd,itr[begin:k],itr[v:end])
itr,_=iterate(el)
end
mdd
end
1.190 μs (43 allocations: 2.23 KiB)
6-element Dictionary{String, Any}
"#FORMAT" │ "EMSA"
"#VERSION" │ "1.0"
"#TITLE" │ "2023-Apr-06"
"#NPOINTS" │ "4096"
"#NCOLUMNS" │ "1"
"#SPECTRUM" │ ""
```
---
### Author:  [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)
#### Post date: [April 7, 2023, 10:47am UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/7 "2023-04-07T10:47:41Z")
I don’t know if it’s already available, but it would be nice to have a multi-tryparse function for a list of dynamically supplied types.
Just to give an idea, like the following hunk
```julia
function mtryparse(str,TS...)
str==""&&return str
i=1
dfrm=DateFormat("y-u-d")
v=tryparse(TS[i],str)
while isnothing(v)&& (i begin
io=IOBuffer(data)
el=eachline(io)
md=startswith("#")
mdd=Dictionary{String, Any}()
itr,_=iterate(el)
while md(itr)
k=findfirst(' ',itr)-1
v=findfirst(':',itr)+2
pv=mtryparse(itr[v:end],Int,Float64,Date)
insert!(mdd,itr[begin:k],pv)
itr,_=iterate(el)
end
mdd
end
6-element Dictionary{String, Any}
"#FORMAT" │ "EMSA"
"#VERSION" │ 1.0
"#TITLE" │ Date("2023-04-06")
"#NPOINTS" │ 4096
"#NCOLUMNS" │ 1
"#SPECTRUM" │ ""
```
---
### Author:  [@Austin\_Weber](https://discourse.julialang.org/u/Austin_Weber)
#### Post date: [April 7, 2023, 3:33pm UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/8 "2023-04-07T15:33:26Z")
Hi Nicholas, it’s funny – I’ve actually watched your YouTube videos on using DTSA-II, so it’s interesting that you would come across my question. I wasn’t aware that you had made a Julia package for working with .msa files, so thank you very much for sharing.
I only started using Julia about a month ago, so I’m still getting used to the syntax and understanding the documentation. Have you made any videos showing how to use the NeXLSpectrum package? I’m mostly interested in making customizable plots, but I’d also like to know how to perform P/B-ZAF corrections and to quantify peak intensity ratios.
I appreciate the help!
---
### Author:  [@NicholasWMRitchie](https://discourse.julialang.org/u/NicholasWMRitchie)
#### Post date: [April 7, 2023, 6:26pm UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/9 "2023-04-07T18:26:14Z")
Austin,
There is documentation here: [Home · NeXLSpectrum.jl](https://pages.nist.gov/NeXLSpectrum.jl/)
In specific, fitting and quantification is documented here: [Fitting K412 (simple API) · NeXLSpectrum.jl](https://pages.nist.gov/NeXLSpectrum.jl/k412refs/)
I haven’t implemented peak-to-background corrections (only φ(ρz)) but, if you’d like to, …
You might find these pages helpful too: [(Image: )Core - Part of the NeXL X-ray Microanalysis Library · NeXLCore](https://pages.nist.gov/NeXLCore.jl/) and [(Image: )MatrixCorrection - Part of the NeXL X-ray Microanalysis Library · NeXLMatrixCorrection.jl](https://pages.nist.gov/NeXLMatrixCorrection.jl/)
---
### Author:  [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)
#### Post date: [April 7, 2023, 7:06pm UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/10 "2023-04-07T19:06:00Z")
Perhaps we should also read the spectrum matrix data in the `Dict` object?
> **Example using DelimitedFiles**
>
> ```julia
> str = """
> #FORMAT : EMSA
> #VERSION : 1.0
> #TITLE : 2023-Apr-06
> #NPOINTS : 4096
> #NCOLUMNS : 1
> #SPECTRUM :
> 0.00, 0.0
> 10.00, 4.0
> 20.00, 2.0
> 30.00, 7.0
> 40.00, 15.0
> 50.00, 21.0
> """
>
> io = IOBuffer(str)
> lines = readlines(io)
> ix = findfirst(x->first(x)!='#', lines) - 1
> meta = split.(chop.(lines[1:ix], head=1, tail=0),":")
> dic = Dict{AbstractString, Any}(strip.(first.(meta)) .=> strip.(last.(meta)))
>
> using DelimitedFiles
> dic["SPECTRUM"] = readdlm(IOBuffer(str), ',', skipstart=ix+1)
>
> ```
---
### Author:  [@Austin\_Weber](https://discourse.julialang.org/u/Austin_Weber)
#### Post date: [April 7, 2023, 8:00pm UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/11 "2023-04-07T20:00:51Z")
Certainly, although I have just been reading the numerical data directly into a data frame like so:
```julia
using CSV, DataFrames
# The dictionary from earlier in the conversation
dic = Dict(strip.(first.(meta)) .=> strip.(last.(meta)))
skip2 = length(keys(dic)) + 1;
data= CSV.read("datafile.msa",
DataFrame,
skipto=skip2,
delim=",",
header=false,
ignorerepeated=true,
footerskip=1);
```
---
### Author:  [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)
#### Post date: [April 7, 2023, 8:12pm UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/12 "2023-04-07T20:12:16Z")
Thank you. May I ask what is the benefit to your work of having such a numeric matrix with the spectrum in a data frame?
---
### Author:  [@Austin\_Weber](https://discourse.julialang.org/u/Austin_Weber)
#### Post date: [April 7, 2023, 8:22pm UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/13 "2023-04-07T20:22:22Z")
I don’t really know if there is a benefit to it, but I find data frames easy to understand and they are straightforward to work with. I’m very new to Julia, so my opinions on the best way to do things are still malleable.
---
### Author:  [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)
#### Post date: [April 7, 2023, 8:46pm UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/14 "2023-04-07T20:46:10Z")
Right, it’s up to you to find your comfort zone.
In that case, have you considered adding the `dic` dictionary to your dataframe `data` as metadata?
Something like:
```julia
metadata!(data, "Meta", dic, style=:note)
metadata(data, "Meta")
```
---
### Author:  [@Austin\_Weber](https://discourse.julialang.org/u/Austin_Weber)
#### Post date: [April 8, 2023, 4:09pm UTC](https://discourse.julialang.org/t/read-data-file-and-create-a-dictionary-from-the-metadata/97200/15 "2023-04-08T16:09:59Z")
I’ll add that to my tool belt, thanks!