# How do you like to save your data and how much customizability do you like/need?
**URL:** https://discourse.julialang.org/t/how-do-you-like-to-save-your-data-and-how-much-customizability-do-you-like-need/129707
**Category:** General Usage
**Tags:** data, io, formatting
**Created:** [June 6, 2025, 4:56pm UTC](https://discourse.julialang.org/t/how-do-you-like-to-save-your-data-and-how-much-customizability-do-you-like-need/129707 "2025-06-06T16:56:41Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@AwesomeQuest](https://discourse.julialang.org/u/AwesomeQuest)
#### Post date: [June 6, 2025, 4:56pm UTC](https://discourse.julialang.org/t/how-do-you-like-to-save-your-data-and-how-much-customizability-do-you-like-need/129707/1 "2025-06-06T16:56:41Z")
I’m making an application that controls a spectrometer (among other things). The program is likely to be saving dozens if not hundreds of spectra, each being about 0.5MB.
There’s also some positional metadata that comes with each measurement.
So I ask: How would you as a researcher prefer to receive this data? With a prefix/suffix or customizable counter in the file name? How should the metadata be included?
Is there a standard file format I should be using like NetCDF?
How would you like the file saving settings to look like / function?
Each measurement is also related to other measurements in runs so should I actually be saving the individual spectra into one big file to make it easier to display and read the bulk data of a run?
I’m guessing I should also include the option to save the data as .csv afterwards but for speed reasons it’s not really practical to save the data as csv in real time.
---
### Author:  [@cshen](https://discourse.julialang.org/u/cshen)
#### Post date: [June 6, 2025, 6:08pm UTC](https://discourse.julialang.org/t/how-do-you-like-to-save-your-data-and-how-much-customizability-do-you-like-need/129707/2 "2025-06-06T18:08:43Z")
I generally like to have data and metadata separated into different files, but not all metadata is the same, I think that metadata that’s needed to correctly interpret the data should be together with the data in the same file possibly. Exactly what depends a bit on the situation and use case though.
---
### Author:  [@Eben60](https://discourse.julialang.org/u/Eben60)
#### Post date: [June 6, 2025, 6:20pm UTC](https://discourse.julialang.org/t/how-do-you-like-to-save-your-data-and-how-much-customizability-do-you-like-need/129707/3 "2025-06-06T18:20:11Z")
The post cited below was written a few years ago, and in the meanwhile there are also other options, but these listed are presumably still valid. Among the listed formats, HDF5 has the advantage of being supported by many specialized softwares (e.g. I know it is supported by Origin and Igor Pro).
> [@Recommended serialization interface in Oct 2020: JLD, JLD2,](https://discourse.julialang.org/t/recommended-serialization-interface-in-oct-2020-jld-jld2/48994/2):
>
> Sometimes its good to have an open community building a ton of alternatives, sometimes it’s not as good. I think this is a place where it hasn’t turned out so good and someone needs to sit down and consolidate it. The summary right now is: Julia’s serialization is the best, but cannot necessarily cross versions. HDF5 is stable and won’t change, but that means it supports the basics (arrays and such), and that’s pretty much what it will do. JLD2 is great and got new maintainers. That might mean…
---
### Author:  [@AwesomeQuest](https://discourse.julialang.org/u/AwesomeQuest)
#### Post date: [June 7, 2025, 10:51am UTC](https://discourse.julialang.org/t/how-do-you-like-to-save-your-data-and-how-much-customizability-do-you-like-need/129707/4 "2025-06-07T10:51:00Z")
Great! I think I’ve settled on HDF5 as the medium since it seems to be supported by most things (even Excel to some extent) and supports basically arbitrary metadata.
Now onto the file names. How much customisation do y’all like to have when choosing auto-generated file names? How do other programs handle that type of thing?
---
### Author:  [@Jake](https://discourse.julialang.org/u/Jake)
#### Post date: [June 8, 2025, 2:35am UTC](https://discourse.julialang.org/t/how-do-you-like-to-save-your-data-and-how-much-customizability-do-you-like-need/129707/5 "2025-06-08T02:35:41Z")
I am by no means an expert, but I would choose between Arrow and HDF5. Probably worth trying both. Each can store metadata. The one place where I liked Arrow is that if you are collecting an unknown amount of data, the Arrow file can keep on growing, you don’t have to set the file size up front. The downside with this is that the metadata for each channel gets clobbered, though the metadata for the file stays intact. So the trick is to include the channel metadata with the file metadata. I have less experience with HDF5
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [June 8, 2025, 2:39pm UTC](https://discourse.julialang.org/t/how-do-you-like-to-save-your-data-and-how-much-customizability-do-you-like-need/129707/6 "2025-06-08T14:39:01Z")
I like to use `clean_file_name()` ( [PetrKryslUCSD/DataDrop.jl: Numbers and matrices and strings stored to disk and retrieved again.](https://github.com/PetrKryslUCSD/DataDrop.jl)): Windows is quite touchy about file names, and I like to store a lot of numerical & other info in the file name.