# Append rows to a JuliaDB table on disk

**URL:** https://discourse.julialang.org/t/append-rows-to-a-juliadb-table-on-disk/44071
**Category:** General Usage
**Created:** [August 1, 2020, 5:03am UTC](https://discourse.julialang.org/t/append-rows-to-a-juliadb-table-on-disk/44071 "2020-08-01T05:03:53Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)
#### Post date: [August 1, 2020, 5:03am UTC](https://discourse.julialang.org/t/append-rows-to-a-juliadb-table-on-disk/44071/1 "2020-08-01T05:03:53Z")

</div>

I have streaming structured data that I’d like to write to disk either row-by-row or with a small amount of buffering.

Is JuliaDB appropriate for this? I ultimately will want to do some data manipulation on the resulting tables using JuliaDB, so it would be nice if I could keep everything in one format. After a brief search, I couldn’t find a way to append to an on-disk table, only rewrite the entire table, which would be extremely inefficient (incoming data will arrive at around 1-2MB/s).

---

<div class="post-metadata">

### Author: ![gkappler](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gkappler/32/11254_2.png) [@gkappler](https://discourse.julialang.org/u/gkappler)
#### Post date: [August 4, 2020, 9:33am UTC](https://discourse.julialang.org/t/append-rows-to-a-juliadb-table-on-disk/44071/2 "2020-08-04T09:33:38Z")

</div>

I refactored the JuliaDB load code a bit ([https://github.com/JuliaData/JuliaDB.jl/pull/365](https://github.com/JuliaData/JuliaDB.jl/pull/365)) to

1. read a chunk in a `nd::NDSparse` and then
2. merge! with `dnd::JuliaDB.DNDSparse`:

```julia
merge!(dnd, nd; output=x.output)

```

Performance is really good on my SSD laptop, at least writing about 1-2 mio rows/sec with 3 Int64 columns.  
If chunks are small, many chunk files will be created, which I `rechunk!` in a finalizing compression.

Also, PR [https://github.com/JuliaData/JuliaDB.jl/pull/288](https://github.com/JuliaData/JuliaDB.jl/pull/288) is discussing a csv approach.
