# Concatenate CSV data into Feather using DataStreams

**URL:** <https://discourse.julialang.org/t/concatenate-csv-data-into-feather-using-datastreams/3690>\
**Category:** Data\
**Tags:** question\
**Created:** [May 13, 2017, 12:23pm UTC](https://discourse.julialang.org/t/concatenate-csv-data-into-feather-using-datastreams/3690 "2017-05-13T12:23:42Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 13, 2017, 12:23pm UTC](https://discourse.julialang.org/t/concatenate-csv-data-into-feather-using-datastreams/3690/1 "2017-05-13T12:23:43Z")

</div>

I have about 400G of data in 15 `.csv.gz` files, overall 2e9 records with 20 columns, same schema in each file. I would like to read all of them and save in a single `.feather` file for further analysis.

I could not figure out how to do this using `DataStreams`, I tried to use `append = true` but file is not appended but overwritten (MWE below).

Also, I am wondering if I can do this directly if the total data would not fit into memory (which is the case), since `Feather.jl` apparently needs to read in the old data before appending. Or would `mmap` bypass this difficulty? If not, should I use an interim step, eg an `SQLite` database? In any case, the MWE is

```julia
using DataStreams
using CSV
using Feather
using DataTables

# data for the MWE
dt = DataTable(A = 1:10, B = 99:-1:90)
CSV.write("/tmp/test1.csv", dt)
CSV.write("/tmp/test2.csv", dt)

# read and copy first file
src1 = CSV.Source("/tmp/test1.csv")
snk = Feather.Sink("/tmp/test.feather", Data.schema(src1))
Data.stream!(src1, snk)

# read and copy second file
src2 = CSV.Source("/tmp/test2.csv")
snk = Feather.Sink("/tmp/test.feather", Data.schema(src2); append = true)
Data.stream!(src2, snk)

# close
Data.close!(snk)

# data is not appended but overwritten
src_all = Feather.Source("/tmp/test.feather")

```

Using `0.6-rc1`, released versions of all packages.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 13, 2017, 12:55pm UTC](https://discourse.julialang.org/t/concatenate-csv-data-into-feather-using-datastreams/3690/2 "2017-05-13T12:55:58Z")

</div>

And this is how I would do it using SQLite, is this the right way?

```julia
using SQLite
using CSV
using DataStreams
using Feather

db = SQLite.DB("/tmp/test.db")

src1 = CSV.Source("/tmp/test1.csv")
SQLite.load(db, "mytable", src1)

src2 = CSV.Source("/tmp/test2.csv")
SQLite.load(db, "mytable", src2; append = true)

src_all = SQLite.Source(db, "SELECT * FROM mytable")
Feather.write("/tmp/test.feather", src_all)

```

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [February 1, 2018, 4:44am UTC](https://discourse.julialang.org/t/concatenate-csv-data-into-feather-using-datastreams/3690/3 "2018-02-01T04:44:24Z")

</div>

Did you figure it out?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 1, 2018, 5:45am UTC](https://discourse.julialang.org/t/concatenate-csv-data-into-feather-using-datastreams/3690/4 "2018-02-01T05:45:00Z")

</div>

Nope, wrote my own library:  
[https://github.com/tpapp/LargeColumns.jl](https://github.com/tpapp/LargeColumns.jl)  
This is amazingly fast for “ingest once, use repeatedly” workflows (the kernel/mmap does the work transparently). Open an issue if I can help with any extension.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [February 1, 2018, 6:00am UTC](https://discourse.julialang.org/t/concatenate-csv-data-into-feather-using-datastreams/3690/5 "2018-02-01T06:00:03Z")

</div>

It says bits types only, so it doesn’t work with strings. Anyway, I finally got it to run using CSV but it’s about 3x slower than R’s datat.table’s (awesome) `fread`.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 1, 2018, 6:50am UTC](https://discourse.julialang.org/t/concatenate-csv-data-into-feather-using-datastreams/3690/6 "2018-02-01T06:50:32Z")

</div>

One could collect strings and `mmap` into them, but that requires a different approach. What I do is that I accumulate strings in a hash table, mapping to an integer, which I then store using the library above. I reclaim it via `IndirectArrays.jl`. This is very efficient, I have far more observations than unique strings.