Write CSV row by row

Specific Domains Data
,
1

I have a computation that calculates rows of data, let’s say they are NamedTuples with the same keys. I would like to emit this into a CSV file without collecting or making it an iterable.

Why?

Why not collect? The data is large. Why not wrap in an iterable? The algorithm is such that it is easier to pass around a sink to which one writes to, than concatenate many layers of iterators.

Pseudocode for what I want:

sink = make_csv_sink(path, schema::Type{<:NamedTuple{colnames}})

emit(sink, a_namedtuple) # checking that colnames match would be nice

close(sink)

When the fields are numbers or unescaped strings, this can be trivially implemented, but if we get into fancier CSV escapes it becomes tricky. This question has been asked before without a solution.

Existing CSV libraries must have this functionality (or the building blocks for it), but it is not exposed.

2

This would be a useful feature. Writing rows directly to a CSV without collecting everything first feels like a common use case, especially for long running computations. It would be nice if the API also validated the schema on each write.

3

There’s CSV.writerow, which is not public unfortunately but seems to be closest to what you’re looking for.

4

Maybe we can make this part more straightforward in Julia? Turning such a “sink-able” algorithm into an iterable. Seems like that’s the right seam to plug in, and useful way beyond CSV writing.

5

Do you have a suggestion on how to approach this? In my mind they are two fundamentally opposed approaches.

One way I could imagine is a Channel, which blocks iterate unless it has stuff in the queue, which can then be closed and then it would return nothing. But it is somewhat convoluted.