Update You should really be using CSV.jl because it performs quite well. I only use Fread.jl for converting data from parquet to feature etc now and not for reading CSVs as CSV.jl is actually really good. Original content Let’s be honest, all we care about is the speed of CSV reading. I think it …

Just from a quick browse of the code, it seems like you use R’s fread to read the table and write it as a feather file, and then use Julia’s native Feather package to read it back in. Just wondering if this back and forth is still faster than CSV.jl and if so, why? What makes the CSV.jl package muc…

I think one potential issue here is that Feather.read doesn’t actually load the data from disc, it just reads the meta-data, and then the data will get loaded from disc when you actually access values. Not sure how these benchmarks were run, but potentially they didn’t include the reading of the dat…

Oh yes the data is just mapped for feather. I have tried to run a group by after though. The performance is decent.

Firstly reading from feather doesn’t actually read. It just maps. Secondly, fread is very mature (10 years of development), so it’s much faster. If Julia has arrow then the data can be accessed quicker without going thru the feather step.

If anyone finds issue with the benchmark in anyway feel free Try it on your own data suggest a way to eliminate the issues Happy to incorporate. I think Fread.jl will die once Julia has a world beating CSV reader which @quinnj is working on.

[ANN] Fread.jl - read CSVs faster with the help of R's {data.table}

Package Announcements

xiaodai October 9, 2019, 2:20pm 2

One more benchmark

CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R

Topic		Replies	Views
My experiences reading CSVs from the Fannie Mae datasets Data performance , csv	62	6563	August 26, 2019
CSV read in is too slow than other language General Usage performance	13	1588	June 21, 2023
Reading Data Is Still Too Slow Data	35	9141	August 2, 2019
CSV Reader / Writer Choices Data	1	765	August 28, 2018
Alternative to DataFrame Readtable to read large data files with headers Data	17	4150	November 12, 2018

[ANN] Fread.jl - read CSVs faster with the help of R's {data.table}

Related topics