I’ve updated to DataFrame v.0.21.2 and Julia ver 1.4.2. Try to load CSV file (9GB), as result 32 GB Ram is not enough. Is it reasonable? Why DataFrames consume so much memory?

whoops, I edited my comment while you were posting yours… see above.

Yes, I was already trying it. I can load data. Thank you I wonder, why it use twiсe more memory? 18gb ram for 9gb file

My guess is the CSV parser reads the whole file into RAM and then parses it… which results in a lot more memory usage, basically linear in the size of the file. The CSVFiles version reads just a bunch of lines at a time, so it only uses a more or less constant size buffer regardless of the file size…

yes, you are right. In any case, right now 18gb ram is using only for one 9gb file. It’s too many I think.

You’re talking about after parsing and just having it in the DataFrames? What happens when you run GC.gc()? Also, what are the column types that load chose? If they’re Any then that could be an issue.

yes, I am taking about DataFrame, after parsing GC.gc() doesn’t help. p.s. during parsing ram is using even more, =))

what are the data types? If it’s a file full of numbers but it decides its columns of Any, then you would expect maybe 64 bits for pointers and 64 bits for floats = twice the ram necessary if it were able to figure out the types for example.

[image] dlakelan: My guess is the CSV parser reads the whole file into RAM and then parses it… I’m 98% sure this is not the case. @quinnj has put in a lot of work to make CSV.jl as fast as possible, and I’m quite certain that stuff is getting parsed on the fly. Whether there’s another explana…

There’s something strange here. Generating a medium-sized dataframe, julia> a = rand(10^7); julia> Base.summarysize(a) 80000040 julia> df = DataFrame(a = a); julia> Base.summarysize(df) 80000624 julia> CSV.write("test.csv", df) "test.csv" …then reading it back, julia> df2 = CSV.read("test.csv…

Why DataFrames v.0.21.2 (julia v1.4.2) requires more memory than the previous version

General Usage Performance

BMval June 11, 2020, 1:02am 11

much of columns is int64.

Oh, I understand. csv has a char, int64 is twice large.!!!

Is it possible to force use Int32, instead of Int64

Topic		Replies	Views
Reading large CSV files New to Julia dataframes , csv	9	620	August 31, 2026
CSV.write("*.txt",DataFrame) ReadOnlyMemoryError() General Usage dataframes	14	1251	January 9, 2020
CSV.read extremely slow wrt readtable Data	14	3835	July 27, 2018
CSV : problem to write big dataframes Data csv	20	3127	May 29, 2023
Failing to import (relatively) large CSV file with Julia and VSC Data performance , csv , arrow	24	1143	September 22, 2024

Why DataFrames v.0.21.2 (julia v1.4.2) requires more memory than the previous version

Related topics