[ANN] Onda.jl: A format for multi-sensor, multi-channel, LPCM-encodable recordings

[sairus7]

@laborg

Triple level caching/access might be helpful for some cases, but for what I am interested in, two layers feel sufficient (chunking the data from (compressed) disk storage into memory).

Third level appears as soon as you copy, say, 1500 points from cached chunks of 1000 points.

@laborg

For my usage annotations wouldn’t be that excessive. I’ve the feeling that if you are talking about this kind of numbers (>100000) the annotations are either outputs of some kind of algorithm or a signal of its own. From my point of view, both shouldn’t be handled as annotations.

Yes, I’m talking about annotations as one of several signal types, known as labels or segmentation. So, they are indeed output of some algorithm. Especially if we are talking about tebadytes of data, because you cannot manually annotate terabytes in a reasonable amount of time. You can only review a small portion of automatically labeled/annotated data.

@laborg

There is already a paragraph describing the reasoning for Onda in comparison to HDF5 (and other formats). GitHub - beacon-biosignals/OndaFormat: A lightweight format for storing and manipulating sets of multi-sensor, multi-channel, LPCM-encodable, annotated, time-series recordings.

I’ts not about HDF5 limitations, because there are different similar formats like Arrow, Zarr, Exdir, TileDB, N5, Z5, etc.

If Onda is a layer above already stored files of different formats (something like file database), then it is more about mapping different formats with software to read from them, mapping data from files to metadata, and taking special attention to metadata structures that are added on top of those files. Here are some thoughts on working with metadata: Do you use some file database with tagging / multiple grouping functionality?