# Best Data format for Tabular Data plus Vector
**URL:** https://discourse.julialang.org/t/best-data-format-for-tabular-data-plus-vector/51315
**Category:** General Usage
**Created:** [December 5, 2020, 7:09pm UTC](https://discourse.julialang.org/t/best-data-format-for-tabular-data-plus-vector/51315 "2020-12-05T19:09:33Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Jake](https://discourse.julialang.org/u/Jake)
#### Post date: [December 5, 2020, 7:09pm UTC](https://discourse.julialang.org/t/best-data-format-for-tabular-data-plus-vector/51315/1 "2020-12-05T19:09:33Z")
Coming from a background in Matlab, where I do mostly signal processing kinds of things, and am considering switching to Julia, I was looking for good alternatives for data storage in Julia. The type of data I would like to store to begin with is:
Measurement Location
Node Number
Node direction
measurement type
Measurement units
Measurement start time
sampling rate Fs
Measurement Comments
Vector containing the time history of the measurements
The vector could have a size of 10^5 - 10^6. The data will then be post-processed, initially in Matlab but the matlab algorithms will probably move to Julia in due time. The processing will consist of STFT, TVDFT, processing a tach signal into a speed map, etc.
In Matlab I can do this with an array of structures or a table.
Browsing through the Julia documentation it does not seem that a core language structure is ideal for this. Doing an initial review of package data structures ([A Tour of the Data Ecosystem in Julia – Traitement de Données](https://quinnj.home.blog/2019/07/21/a-tour-of-the-data-ecosystem-in-julia/)). At this point it is not clear to me whether a Dataframe, Table or JuliaDB will accept the vector in addition to the tabular data for each channel collected.
Once collected, it may make sense to store the data in a matlab file. What are the other good alternatives for storing this type of data? Note that there can be large amounts of data, so smaller file sizes are preferable.
Some guidance on the data structure and file storage will be appreciated. Thanks in advance.
---
### Author:  [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)
#### Post date: [December 5, 2020, 7:33pm UTC](https://discourse.julialang.org/t/best-data-format-for-tabular-data-plus-vector/51315/2 "2020-12-05T19:33:11Z")
Depending on how many modalities/sample rates you’re handling, I’d imagine an array of structs or table wouldn’t be ideal either 🙂
Perhaps the most purpose-built format I’ve seen is [https://github.com/beacon-biosignals/OndaFormat](https://github.com/beacon-biosignals/OndaFormat). There’s a native Julia SDK at [https://github.com/beacon-biosignals/Onda.jl](https://github.com/beacon-biosignals/Onda.jl). Matlab files are either a basic flat array + metadata or dialect of HDF5 anyhow, so you could roll something custom on top of HDF5.jl or Zarr.jl as well. A couple of folks from my lab were recently working on [multi-modality format for health data](https://conduitlab.github.io/ccdef/), might be worth a look if that’s what you’re working with.
---
### Author:  [@hendri54](https://discourse.julialang.org/u/hendri54)
#### Post date: [December 5, 2020, 9:57pm UTC](https://discourse.julialang.org/t/best-data-format-for-tabular-data-plus-vector/51315/3 "2020-12-05T21:57:49Z")
Why not define a custom type that holds a DataFrame (for the tabular data) and a Vector{Float64}, for example?
As for saving the data to a file, my suggestion would be to save two separate files for robustness. Robustly storing user defined types is still a bit iffy in Julia.
If you store the tabular data in a DataFrame [JDF.jl](https://github.com/xiaodaigh/JDF.jl) is a good option. For the Vector{Float} there are many options. I like [JLD2](https://github.com/JuliaIO/JLD2.jl), but that’ s just my preference. More info can be found by searching Discourse for threads on serialization.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [December 6, 2020, 2:22pm UTC](https://discourse.julialang.org/t/best-data-format-for-tabular-data-plus-vector/51315/4 "2020-12-06T14:22:08Z")
> [@Jake](#):
>
> Vector containing the time history of the measurements
Because of the length of the vector, especially if it is variable, a dataframe is not ideal for this. You could make it [tidy](https://vita.had.co.nz/papers/tidy-data.pdf) for some applications, but then it may be wasteful for repeated fields.
I would decouple the choice of _Julia data type_ (for which I would just start with a `struct`) from the _externalized representation_, where you can experiment with various options — eg JSON may work as a portable and lightweight — if somewhat verbose — option, or HDF5 if speed is important.