# Streaming CSV Reader from IO Type which does not load all Data into memory

**URL:** https://discourse.julialang.org/t/streaming-csv-reader-from-io-type-which-does-not-load-all-data-into-memory/51050
**Category:** General Usage
**Created:** [December 1, 2020, 3:37pm UTC](https://discourse.julialang.org/t/streaming-csv-reader-from-io-type-which-does-not-load-all-data-into-memory/51050 "2020-12-01T15:37:09Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![altre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/altre/32/17036_2.png) [@altre](https://discourse.julialang.org/u/altre)
#### Post date: [December 1, 2020, 3:37pm UTC](https://discourse.julialang.org/t/streaming-csv-reader-from-io-type-which-does-not-load-all-data-into-memory/51050/1 "2020-12-01T15:37:09Z")

</div>

I am looking for a way to read from a stream of CSV Rows, i.e. from an `IO` (or LibuvStream) type _without_ reading the whole table into memory.

This currently works e.g. with CSV.jl’s CSV.Rows when reading a file since it is read as a memory mapped file.

However I would like to read row by row from an arbitrary `IO` input (such as streaming from s3). Currently all packages seem to call `Base.read(io)` in these cases.

---

<div class="post-metadata">

### Author: ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)
#### Post date: [December 1, 2020, 3:50pm UTC](https://discourse.julialang.org/t/streaming-csv-reader-from-io-type-which-does-not-load-all-data-into-memory/51050/2 "2020-12-01T15:50:14Z")

</div>

Couldn’t you read the stream row by row into `io` and then periodically pass that `io` to CSV.jl for parsing? Sort of a roll your own batched processing approach

---

<div class="post-metadata">

### Author: ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)
#### Post date: [December 1, 2020, 3:56pm UTC](https://discourse.julialang.org/t/streaming-csv-reader-from-io-type-which-does-not-load-all-data-into-memory/51050/3 "2020-12-01T15:56:08Z")

</div>

Also it may be worth pointing out that if you’re reading over a network connection, you will probably get better results by reading a fairly large chunk at a time rather than trying to go like by line.
