# Why DataFrames v.0.21.2 (julia v1.4.2) requires more memory than the previous version

**URL:** <https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171>\
**Category:** Performance\
**Tags:** dataframes\
**Created:** [June 10, 2020, 11:55pm UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171 "2020-06-10T23:55:42Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [June 10, 2020, 11:55pm UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/1 "2020-06-10T23:55:42Z")

</div>

I’ve updated to DataFrame v.0.21.2 and Julia ver 1.4.2.

Try to load CSV file (9GB), as result 32 GB Ram is not enough.

Is it reasonable? Why DataFrames consume so much memory?

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [June 11, 2020, 12:18am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/2 "2020-06-11T00:18:11Z")

</div>

Which CSV reader Package are you using? I ask because @davidanthoff specifically mentioned in some of his YouTube tutorials that the Queryverse CSVFiles doesn’t pre-read the whole file. so

```julia
using CSVFiles
load("my.csv") |> DataFrame() 

```

should use a reasonable quantity of buffering. I don’t know if that’s the case for CSV.

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [June 11, 2020, 12:19am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/3 "2020-06-11T00:19:12Z")

</div>

I use CSV reader.

What do you suggest to use?

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [June 11, 2020, 12:20am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/4 "2020-06-11T00:20:09Z")

</div>

whoops, I edited my comment while you were posting yours… see above.

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [June 11, 2020, 12:24am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/5 "2020-06-11T00:24:48Z")

</div>

Yes, I was already trying it. I can load data. Thank you  
I wonder, why it use twiсe more memory? 18gb ram for 9gb file

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [June 11, 2020, 12:26am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/6 "2020-06-11T00:26:32Z")

</div>

My guess is the CSV parser reads the whole file into RAM and then parses it… which results in a lot more memory usage, basically linear in the size of the file. The CSVFiles version reads just a bunch of lines at a time, so it only uses a more or less constant size buffer regardless of the file size. This is just a guess though.

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [June 11, 2020, 12:31am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/7 "2020-06-11T00:31:04Z")

</div>

yes, you are right.

In any case, right now 18gb ram is using only for one 9gb file. It’s too many I think.

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [June 11, 2020, 12:32am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/8 "2020-06-11T00:32:05Z")

</div>

You’re talking about after parsing and just having it in the DataFrames? What happens when you run GC.gc()?

Also, what are the column types that load chose? If they’re Any then that could be an issue.

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [June 11, 2020, 12:51am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/9 "2020-06-11T00:51:29Z")

</div>

yes, I am taking about DataFrame, after parsing

GC.gc() doesn’t help.

p.s. during parsing ram is using even more, =))

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [June 11, 2020, 1:00am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/10 "2020-06-11T01:00:17Z")

</div>

what are the data types? If it’s a file full of numbers but it decides its columns of Any, then you would expect maybe 64 bits for pointers and 64 bits for floats = twice the ram necessary if it were able to figure out the types for example.

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [June 11, 2020, 1:02am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/11 "2020-06-11T01:02:00Z")

</div>

much of columns is int64.

Oh, I understand. csv has a char, int64 is twice large.!!!

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [June 11, 2020, 1:18am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/12 "2020-06-11T01:18:47Z")

</div>

> [@dlakelan](#):
>
> My guess is the CSV parser reads the whole file into RAM and then parses it…

I’m 98% sure this is not the case. @quinnj has put in a lot of work to make CSV.jl as fast as possible, and I’m quite certain that stuff is getting parsed on the fly. Whether there’s another explanation for the memory use, I don’t know.

---

<div class="post-metadata">

**Author:** ![stillyslalom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stillyslalom/32/45687_2.png) [@stillyslalom](https://discourse.julialang.org/u/stillyslalom)\
**Post date:** [June 11, 2020, 1:32am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/13 "2020-06-11T01:32:50Z")

</div>

There’s something strange here. Generating a medium-sized dataframe,

```julia
julia> a = rand(10^7);

julia> Base.summarysize(a)
80000040

julia> df = DataFrame(a = a);

julia> Base.summarysize(df)
80000624

julia> CSV.write("test.csv", df)
"test.csv"

```

…then reading it back,

```julia
julia> df2 = CSV.read("test.csv")
10000000×1 DataFrame
│ Row │ a │
│ │ Float64 │
├──────────┼────────────┤
│ 1 │ 0.150381 │
│ 2 │ 0.159632 │
│ 3 │ 0.869726 │
 ...

julia> Base.summarysize(df2.a)
281297536

```

Why does the dataframe’s memory footprint inflate by 3.5x between being written and read from disk?

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [June 11, 2020, 2:03am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/14 "2020-06-11T02:03:33Z")

</div>

CSVFiles helps me to load data. Unfortunately, CSV can’t help me.

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [June 11, 2020, 2:29am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/15 "2020-06-11T02:29:02Z")

</div>

This is an interesting observation and may merit a topic on its own (unless the OP’s problem is similar and willing to use this as a MWE in his original post).

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [June 11, 2020, 2:29am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/16 "2020-06-11T02:29:31Z")

</div>

> [@stillyslalom](#):
>
> `julia> a = rand(10^7);`

This problem doesn’t occur when you use `CSV.File("test.csv") |> DataFrame`.

So i guess this is technically fixed.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [June 11, 2020, 5:17am UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/17 "2020-06-11T05:17:07Z")

</div>

> [@BMval](#):
>
> Try to load CSV file (9GB), as result 32 GB Ram is not enough.

You might wanna try to read the file in chunks using [DataConvenience.jl](https://github.com/xiaodaigh/DataConvenience.jl)

```julia
# read all column as String
for chunk in CsvChunkIterator(filepath)
  # df is a DataFrame where each column is String
  # do something to df
end

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [June 11, 2020, 3:21pm UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/18 "2020-06-11T15:21:01Z")

</div>

> [@BMval](#):
>
> CSVFiles helps me to load data. Unfortunately, CSV can’t help me.

Could you give us more information for how `CSV` fails? Have you tried using `CSV.File` rather than `CSV.read`?

---

<div class="post-metadata">

**Author:** ![BMval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bmval/32/7647_2.png) [@BMval](https://discourse.julialang.org/u/BMval)\
**Post date:** [June 15, 2020, 7:55pm UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/19 "2020-06-15T19:55:19Z")

</div>

It’s a table with 2000 columns 0 and 1.  
I didn’t try CSV.Files.  
I always used CSV.read.

Right now I use CSVFiles.

---

<div class="post-metadata">

**Author:** ![uwechsler](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uwechsler/32/5223_2.png) [@uwechsler](https://discourse.julialang.org/u/uwechsler)\
**Post date:** [June 15, 2020, 9:29pm UTC](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171/20 "2020-06-15T21:29:29Z")

</div>

Sorry for repeating the question.

But can you try this

> [@pdeffebach](#):
>
> `CSV.File("test.csv") |> DataFrame` .

instead of `CSV.read("test.csv")`?

If I recall correctly, @quinnj mentioned in a post that `CSV.read` is going to be deprecated soon and should not be used anymore.

[Next page](https://discourse.julialang.org/t/why-dataframes-v-0-21-2-julia-v1-4-2-requires-more-memory-than-the-previous-version/41171.md?page=2)