I also saw such problems that need a simple preprocessing of a file before feeding it to CSV or another reader library, and also had to resort to similar workarounds.
It should be possible to implement general functions like map_stream_bytes(::Function, io) -> io, map_stream_chars, map_stream_lines (and filter_... as well): they would take and return IO streams without reading all the content to memory at once. Or maybe something like iostream_to_iterator_of_chars/iostream_from_iterator_of_chars is even better.
Didn’t go as far as implementing these functions myself though.
1 Like