I have a large amount of data I am loading from an API and I would like to write the output of each call to a “row” of a file in order not to keep it in RAM/ in case the program errors. I have been using the Arrow.jl package for data reading/writing needs recently and really like it. However, I have…

Write data to Arrow file row by row

danielw2904 May 31, 2022, 4:53am 4

I ended up using the jsonlines format instead. If the file ends up too large I use it with GZip.jl

Saving data a scan at a time and reading as a contiguous block

Topic		Replies	Views
Writing Arrow files by column Performance	1	160	May 8, 2024
Help with Arrow.jl and size of files Data question , arrow	23	1799	October 21, 2022
Serialization format allow incremental write to file General Usage serialization	10	378	March 12, 2023
Cannot read file written by Arrow.jl in Python Data python , arrow	3	541	April 25, 2023
Arrow.write on NamedTuple General Usage	2	401	February 3, 2021