# Saving data a scan at a time and reading as a contiguous block

**URL:** <https://discourse.julialang.org/t/saving-data-a-scan-at-a-time-and-reading-as-a-contiguous-block/101691>\
**Category:** General Usage\
**Created:** [July 17, 2023, 1:00pm UTC](https://discourse.julialang.org/t/saving-data-a-scan-at-a-time-and-reading-as-a-contiguous-block/101691 "2023-07-17T13:00:41Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![Jake](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jake/32/46007_2.png) [@Jake](https://discourse.julialang.org/u/Jake)\
**Post date:** [July 17, 2023, 9:20pm UTC](https://discourse.julialang.org/t/saving-data-a-scan-at-a-time-and-reading-as-a-contiguous-block/101691/3 "2023-07-17T21:20:25Z")

</div>

Looking at some more background information based on the previous comment, I searched for append to a file. I came across this [topic](https://discourse.julialang.org/t/write-data-to-arrow-file-row-by-row/59165/4) which is most useful. This led me to the section in the [Arrow manual](https://arrow.apache.org/julia/dev/reference/#Arrow.Writer) that describes the writer function. An example is

```julia
julia> writer = open(Arrow.Writer, "test.arrow");

julia> for j = 1:400:100_000
       A = Tables.table(transpose(reshape(1:400, 4, 100)) .+ j)
       Arrow.write(writer, A)
       end
julia> close(writer)

julia> table = Arrow.Table("test.arrow")
Arrow.Table with 24900 rows, 4 columns, and schema:
 :Column1 Int64
 :Column2 Int64
 :Column3 Int64
 :Column4 Int64

julia> table[1][1:3]
3-element Vector{Int64}:
  2
  6
 10

```

This successfully writes out data from 4 channels sampled at 51.2 kHz in 1 second chunks. That works in both Float32 and Float64 using the SD card on PI for a rate of 16 Mbits/s. Sometimes on the first invocation it gives a buffer overrun error. I assume this is due to JIT compilation.

Now my wish is to also save some structures to the same file to contain the test and channel metadata. I see that there is an [Arrow.struct](https://arrow.apache.org/julia/dev/reference/#Arrow.Struct) but no example on how to save it, let alone save it in conjunction with my table data.

Is this possible?

---

_[View the full topic](https://discourse.julialang.org/t/saving-data-a-scan-at-a-time-and-reading-as-a-contiguous-block/101691)._