# Concatenate csv files without loading them

**URL:** <https://discourse.julialang.org/t/concatenate-csv-files-without-loading-them/35527>\
**Category:** General Usage\
**Tags:** question, csv\
**Created:** [March 4, 2020, 3:22pm UTC](https://discourse.julialang.org/t/concatenate-csv-files-without-loading-them/35527 "2020-03-04T15:22:48Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [March 4, 2020, 3:22pm UTC](https://discourse.julialang.org/t/concatenate-csv-files-without-loading-them/35527/1 "2020-03-04T15:22:48Z")

</div>

I’m wondering what the best way would be to concatenate multiple CSV files – all of which contain the same header row (they are compatible) – into one CSV file.

I know I can do something like (taken from @c42f’s [one-liner](https://discourse.julialang.org/t/fun-one-liners/28352/3)):

```julia
vcat(CSV.read.(file_names)...) |> CSV.write("one_big_file.csv")

```

But since I never need the data loaded, is there a faster way that avoids fully parsing the data…?

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 4, 2020, 6:32pm UTC](https://discourse.julialang.org/t/concatenate-csv-files-without-loading-them/35527/2 "2020-03-04T18:32:27Z")

</div>

See my answer [here](https://discourse.julialang.org/t/removing-the-first-line-in-a-text-file/35521/6) on removing the first line from a text file. Don’t parse the data, just copy it blindly to a new file, but skip the first (header) line for everything but the first file.

---

<div class="post-metadata">

**Author:** ![scottedwards2000](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottedwards2000/32/31908_2.png) [@scottedwards2000](https://discourse.julialang.org/u/scottedwards2000)\
**Post date:** [February 12, 2022, 4:11pm UTC](https://discourse.julialang.org/t/concatenate-csv-files-without-loading-them/35527/3 "2022-02-12T16:11:02Z")

</div>

Very handy @stevengj ! But we still have the issue of merging all these files without loading them into memory, right @yakir12 ? What’s the best way to solve that?

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [February 12, 2022, 5:04pm UTC](https://discourse.julialang.org/t/concatenate-csv-files-without-loading-them/35527/4 "2022-02-12T17:04:47Z")

</div>

Steve’s nice solution using: `Iterators.drop(eachline(input), 1)` reads and writes the files line by line.

I have just tried it to merge two 47GB files and it seemed to work, taking ~10 min on my PC laptop.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 12, 2022, 11:33pm UTC](https://discourse.julialang.org/t/concatenate-csv-files-without-loading-them/35527/5 "2022-02-12T23:33:19Z")

</div>

> [@rafael.guerra](#):
>
> Steve’s nice solution using: `Iterators.drop(eachline(input), 1)` reads and writes the files line by line.

After you read the first line to strip off the header, it will be a lot faster to read the file in chunks, as in this example: [How to obtain the result of a diff between 2 files in a loop? - #4 by stevengj](https://discourse.julialang.org/t/how-to-obtain-the-result-of-a-diff-between-2-files-in-a-loop/23784/4)

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [February 12, 2022, 11:58pm UTC](https://discourse.julialang.org/t/concatenate-csv-files-without-loading-them/35527/6 "2022-02-12T23:58:48Z")

</div>

> [@stevengj](#):
>
> to read the file in chunks

In the example linked, the chunks seem to be `32768 bytes` long. Why this value?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [February 13, 2022, 12:03am UTC](https://discourse.julialang.org/t/concatenate-csv-files-without-loading-them/35527/7 "2022-02-13T00:03:38Z")

</div>

It’s a power of 2 and a common size for L1 cache.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [February 13, 2022, 12:09am UTC](https://discourse.julialang.org/t/concatenate-csv-files-without-loading-them/35527/8 "2022-02-13T00:09:19Z")

</div>

Thanks Oscar.  
In my laptop I see this:

![L1_L2_L3_cache](https://global.discourse-cdn.com/julialang/original/3X/b/e/be790f3bf2f7a15d9f10cbb1479717a84c81e2e4.png)

Does it mean that I should use a chunk size `= 2^18 = 262144 (< 320 KB)`?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [February 13, 2022, 12:19am UTC](https://discourse.julialang.org/t/concatenate-csv-files-without-loading-them/35527/9 "2022-02-13T00:19:54Z")

</div>

it probably will be a minor difference, but feel free to do some benchmarks…

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [February 14, 2022, 6:42pm UTC](https://discourse.julialang.org/t/concatenate-csv-files-without-loading-them/35527/10 "2022-02-14T18:42:58Z")

</div>

@Oscar_Smith, fyi, I’ve observed ~22% speed gains on my laptop (_with L1 cache = 320 KB_) when using a chunk size of `262_144` bytes instead of `32_768` bytes.

In any case, doing it by chunks seemed much faster than doing it line by line. The code used to merge 2 x 37 GB csv files was adapted from Steve’s original and is provided below.

> **Original code by @stevengj (adapted)**
>
> ```julia
> open("merge_two_37GB.csv", "w") do output
> isfirst = true
> for file in files
> open(file, "r") do input
> if isfirst
> println(output, readline(input))
> isfirst=false
> else
> readline(input) # read repeated CSV header but
> println(output) # print carriage return only
> end
> buf = Vector{UInt8}(undef, 262144) # L1 cache: 262144 => 252 s; 32768 => 323 s
> while !eof(input)
> nb = readbytes!(input, buf)
> write(output, view(buf,1:nb))
> end
> end
> end
> end
> 
> ```

---

<div class="post-metadata">

**Author:** ![scottedwards2000](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottedwards2000/32/31908_2.png) [@scottedwards2000](https://discourse.julialang.org/u/scottedwards2000)\
**Post date:** [April 13, 2022, 9:56pm UTC](https://discourse.julialang.org/t/concatenate-csv-files-without-loading-them/35527/11 "2022-04-13T21:56:20Z")

</div>

This approach worked like a charm!!! so fast! thanks!