Data structure for visualizing and scaling


New julia user here coming from MATLAB background and enjoying things so far.

I have been playing with ImPlot.jl + CImGui and am inspired to create a log viewer and realtime visualizer for a system I’ve inherited.

I have a CSV-like file with hierarchical data and several other logfiles to ingest and visualize. The logfiles will become annotations for faults and such. The CSV-like file has 7 rows per point of time reported at 1/10 hz: first row has 6 elements (timestamp, int, int, 16-bit hex int, int, int) and the remaining six rows have 10 elements (timestamp, int, int, int, int, int, int, int, int, int). Hierarchically, each pair of the 6 rows represents a battery with one row representing a cell in that battery. I usually unroll that 16 bit hex into bits for digital plotting to the stripchart. The total elements including even those not needed plus an index column usually runs 83 elements long.

The data can be interrupted by lines with a single element, a bunch of null characters, and non-CSV data for certain faults.

Previously I had a perl script parse the file and wrap all the lines together into the long-lined 83 element CSV (proper) and then CSV.file() read it in.

The logs account for a 24 hour period, so I may add some state machinery to indicate Start, Run, Stop information to the strip chart and stitch together incomplete sets that span the 24 hr day.

What kind of data_structures in memory and on disk should I be looking at and playing with?
For example, GitHub - JuliaIO/JLD2.jl: HDF5-compatible file format in pure Julia for storing/caching the ingested information worth saving and appending with new datasets as they come in.

Joe Gorse