I have a large amount of data I am loading from an API and I would like to write the output of each call to a “row” of a file in order not to keep it in RAM/ in case the program errors. I have been using the Arrow.jl package for data reading/writing needs recently and really like it. However, I have not found a way to append rows to an Arrow file and was wondering if that is possible?
Here is my current example that does not work:
using Arrow, Tables dat = [(a=1, b=2), (a=3,b=4)] open("test.feather", "a+") do io for row in dat Arrow.write(io, [row]) end end Arrow.Table("test.feather") |> Tables.rowtable