Hello, I’ve a large number of time series as result of my simulations. As today the dataset is made of ~1e7 time series (100 points each), stored on disk in .csv files (each one containing 5 of them). I’ll have more (10x more) in the future and I need to find a good way of exploring the dataset. I…

Consider using a binary format designed for scientific data, e.g. HDF5.jl. 1e8 datasets of 100 points each, in double precision, is only about 80GB (and you might also consider single precision to save 50%), which is easily possible to store in a single file. In HDF5 you can store this as a single…

I would use a SQL database. I’ve found them good even for relatively small use cases, systems like Postgres are extremely well-tested and solve many tricky problems like upserts. It often makes sense to default to using a relational db until you have a specific use case that would benefit from somet…

Similar experience here with HDF5 files, have stored 100million+ records in a single file with 5million/sec average read time (of slices). This was way back in 2016 on an older Intel chip with data stored on a SSD. I would have recommended SQLite also if it was not for 1e8 datasets, I don’t know ho…

As for the databases, you may also consider QuestDB (TimeSeries DB, Java + Rust). It is an open-source alternative to kdb+ (the financial time series standard) and is a pure joy to use (easy setup, convenient, and performant). Basic charting can be done in just a second from within the GUI, or, as i…

For me SQLite (wrapped in Julia) handles ≈1.8e9 records in a single ≈400 GB file fine, definitely much faster than .csv. But I have not benchmarked yet against HDF5, which could be a notch faster regarding the raw reading. What does “exploring” these 1e8 time series mean? If it is simulated data th…

My QuestDB instance is currently quite small, consisting of 10,807 files totaling 4.3 GB. It runs on a cloud VM with 3 ARM64 cores and 16 GB of RAM inside a Podman container. BeeGFS is served by 6 VMs with mixed bandwidth ranging from 0.5 Gbps to 4 Gbps. Its storage is located on an XFS filesystem, …

[image] alemelis: As today the dataset is made of ~1e7 time series (100 points each), stored on disk in .csv files (each one containing 5 of them). I’ll have more (10x more) in the future and I need to find a good way of exploring the dataset. If the data is this regular you could also just c…

Thank you all for the replies. HDF5 looks interesting and the idea closely match my current folder structure. However, I don’t have enough RAM to hold the entire dataset, would that be a problem? This is why I was thinking of a SQL database Regarding the post processing, I don’t know what I’ll do …

To use HDF5, you don’t need to be able to hold all data or even an entire dataset in memory, you can read smaller pieces. Personally, I like HDF5 to hold data in an organized manner, and I find it fairly easy and convenient to use, and the performance is good enough.

Storing lots of time series

Specific Domains Data

Topic		Replies	Views
How do you store your data before and after processing with Julia? Data	35	7478	March 1, 2021
What software/packages/databases to work with 100GB datasets? Offtopic	12	3665	December 18, 2018
TimeSeries Database General Usage question , data , time-series , csv , database	8	786	February 21, 2023
Best practices for local exploratory data analysis Data	15	1728	October 16, 2024
Storing huge amount of data efficiently Performance performance , jld2 , numerics , io , arrow	15	3003	February 24, 2023

Storing lots of time series

Related topics