I have a large amount of data I am loading from an API and I would like to write the output of each call to a “row” of a file in order not to keep it in RAM/ in case the program errors. I have been using the Arrow.jl package for data reading/writing needs recently and really like it. However, I have…

Write data to Arrow file row by row

danielw2904 May 31, 2022, 4:53am 4

I ended up using the jsonlines format instead. If the file ends up too large I use it with GZip.jl

Saving data a scan at a time and reading as a contiguous block

Topic		Replies	Views
Writing Arrow files by column Performance	1	175	May 8, 2024
Serialization format allow incremental write to file General Usage serialization	10	430	March 12, 2023
Arrow.write on NamedTuple General Usage	2	420	February 3, 2021
An example of Apache Arrow file? Data arrow	7	2937	April 22, 2021
[ANN] Arrow.jl 0.3 Release Data arrow	21	3219	March 16, 2021