Append rows to a JuliaDB table on disk

General Usage
1

I have streaming structured data that I’d like to write to disk either row-by-row or with a small amount of buffering.

Is JuliaDB appropriate for this? I ultimately will want to do some data manipulation on the resulting tables using JuliaDB, so it would be nice if I could keep everything in one format. After a brief search, I couldn’t find a way to append to an on-disk table, only rewrite the entire table, which would be extremely inefficient (incoming data will arrive at around 1-2MB/s).

2

I refactored the JuliaDB load code a bit (https://github.com/JuliaData/JuliaDB.jl/pull/365) to

  1. read a chunk in a nd::NDSparse and then
  2. merge! with dnd::JuliaDB.DNDSparse:
merge!(dnd, nd; output=x.output)

Performance is really good on my SSD laptop, at least writing about 1-2 mio rows/sec with 3 Int64 columns.
If chunks are small, many chunk files will be created, which I rechunk! in a finalizing compression.

Also, PR https://github.com/JuliaData/JuliaDB.jl/pull/288 is discussing a csv approach.