I have a large amount of data I am loading from an API and I would like to write the output of each call to a “row” of a file in order not to keep it in RAM/ in case the program errors. I have been using the Arrow.jl package for data reading/writing needs recently and really like it. However, I have…

Write data to Arrow file row by row

danielw2904 May 31, 2022, 4:53am 4

I ended up using the jsonlines format instead. If the file ends up too large I use it with GZip.jl

Saving data a scan at a time and reading as a contiguous block

Topic		Replies	Views
Writing Arrow files by column Performance	1	170	May 8, 2024
Serialization format allow incremental write to file General Usage serialization	10	398	March 12, 2023
Append a `DataFrame` to a partition of an existing `ArrowTable` without creating a new `ArrowTable`? New to Julia question , data , dataframes , arrow	7	568	April 21, 2023
Arrow stream usage clarification Data dataframes , arrow	10	1563	July 17, 2023
Arrow.write on NamedTuple General Usage	2	415	February 3, 2021