# CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R
**URL:** https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974
**Category:** General Usage
**Tags:** announcement
**Created:** [June 24, 2020, 12:12pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974 "2020-06-24T12:12:07Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@oheil](https://discourse.julialang.org/u/oheil)
#### Post date: [June 24, 2020, 12:12pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/1 "2020-06-24T12:12:07Z")
> **[Programmiersprachen-Benchmark: Julia liest Daten viel schneller als Python und R](https://www.heise.de/news/Programmiersprachen-Benchmark-Julia-liest-Daten-viel-schneller-als-Python-und-R-4793906.html)**
>
> Beim jüngsten Release von Julia stand Performance im Fokus, und in einem Benchmark hat der CSV-Parser der Sprache besonders gut abgeschnitten.
This is huge!
For non germans it is about recent performance boosts in Julia 1.4 and refers mainly to the following Blog
[CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R - JuliaHub](https://juliahub.com/blog/2020/06/fast-csv/) (blog url updated)
---
### Author:  [@robsmith11](https://discourse.julialang.org/u/robsmith11)
#### Post date: [June 24, 2020, 12:43pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/2 "2020-06-24T12:43:35Z")
Very impressive results, especially the multi-threaded scaling.
Still… I can’t help but think the title could have been a bit more fair to R and Python. A 10-20x advantage isn’t a fair summary of the results.
---
### Author:  [@oheil](https://discourse.julialang.org/u/oheil)
#### Post date: [June 24, 2020, 12:46pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/3 "2020-06-24T12:46:34Z")
Somehow true, as it depends strongly on the use case and the usage of threads.
I am not sure if the average data analyst, who uses R now, will see any performance increase while playing around with Julias CSV.jl.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [June 24, 2020, 12:49pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/4 "2020-06-24T12:49:55Z")
I can’t help thinking that this framing undersells Julia — the real story here is that _pure Julia_ CSV reading beats _highly optimized C libraries_ that are popular to call from Python and R.
(Though this is explained in the second paragraph.)
---
### Author:  [@oheil](https://discourse.julialang.org/u/oheil)
#### Post date: [June 24, 2020, 12:54pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/5 "2020-06-24T12:54:45Z")
> [@stevengj](#):
>
> _pure Julia_ CSV reading beats _highly optimized C libraries_
This is emphasized in the german article (don’t know if you can read it, so I just tell it 🙂 ).
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [June 24, 2020, 2:03pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/6 "2020-06-24T14:03:14Z")
I can’t tell the R and Julia colours apart.
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [June 24, 2020, 2:04pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/7 "2020-06-24T14:04:35Z")
The Fannie Mae perf dataset contains the largest files. I wish that was tested instead of just testing the Acquisition files.
---
### Author:  [@tbeason](https://discourse.julialang.org/u/tbeason)
#### Post date: [June 24, 2020, 2:51pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/8 "2020-06-24T14:51:37Z")
I wish absolute timings would have been reported somewhere.
What is great is that the differences become apparent even for smaller thread counts. Sure, some people might be able to spin up 16+ threads, but most consumers will not have these high performance machines, and instead will have 4/6/8 threads.
Also ditto that the color scheme chosen for those plots is absolutely terrible.
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [June 24, 2020, 2:53pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/9 "2020-06-24T14:53:14Z")
> [@xiaodai](#):
>
> The Fannie Mae perf dataset contains the largest files. I wish that was tested instead of just testing the Acquisition files.
To be fair for sometimes fread is still faster than CSV.jl on my machine (6 cores). But CSV.jl has gotten to the point where I think it’s viable to do data manipulation in Julia instead.
---
### Author:  [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)
#### Post date: [June 24, 2020, 2:56pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/10 "2020-06-24T14:56:13Z")
This post was great! The one thing I wish it had was some memory analysis. I assume Julia uses more, but I would love to be wrong.
---
### Author:  [@lungben](https://discourse.julialang.org/u/lungben)
#### Post date: [June 24, 2020, 3:52pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/11 "2020-06-24T15:52:09Z")
Looks great!
Did the benchmark measure reading the csv into a DataFrame, i.e. `CSV.File("test_data.csv") |> DataFrame`?
However, the first time executing this command takes quite long for me due to compilation time.
I am not sure in which cases recompilation is required, probably if the column (types) of the csv files are different?
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [June 24, 2020, 4:07pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/12 "2020-06-24T16:07:59Z")
> [@lungben](#):
>
> takes quite long for me due to compilation time.
The main branch of CSV works quite well for me.
---
### Author:  [@Juan](https://discourse.julialang.org/u/Juan)
#### Post date: [June 24, 2020, 6:18pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/13 "2020-06-24T18:18:34Z")
Most benchmarks I’ve seen before showed that data.table was much faster.
> [@\[ANN\] Fread.jl - read CSVs faster with the help of R's {data.table}](https://discourse.julialang.org/t/ann-fread-jl-read-csvs-faster-with-the-help-of-rs-data-table/29681/2):
>
> One more benchmark [image]
Has the situations changed that much? Or they have just used different tests?
---
### Author:  [@lungben](https://discourse.julialang.org/u/lungben)
#### Post date: [June 24, 2020, 6:33pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/14 "2020-06-24T18:33:15Z")
My test case is
```julia
using CSV, DataFrames, PyCall, BenchmarkTools
pd = pyimport("pandas")
download("https://nyc-tlc.s3.amazonaws.com/trip+data/green_tripdata_2019-12.csv",
"test_data.csv")
@time df = CSV.File("test_data.csv") |> DataFrame # including compilation
@btime df = CSV.File("test_data.csv") |> DataFrame
@btime pydf = pd.read_csv("test_data.csv")
```
The 1st csv reading takes 22s for me (on a quite weak machine), but each consequtive reading only 400 ms.
For comparison, Pandas takes 2s.
I tested on Julia 1.4.1 using the most recent versions of CSV.jl and DataFrames.jl.
---
### Author:  [@affans](https://discourse.julialang.org/u/affans)
#### Post date: [June 24, 2020, 6:37pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/15 "2020-06-24T18:37:38Z")
I am still under the impression that `fread` is faster (in terms of “feel”). Let me explain. First, loading `library(data.table)` is much faster than `using CSV, DataFrames`. Although it makes sense since one is loading compiled libraries, the other has to either precompile (or compile at first run).
The second reason i think `fread` is faster is simply the printing of the results back in the console. I don’t know why, but loading a big dataframe in the REPL is often “laggy” and “choppy” for me, but `fread` reads and prints almost instantly. Maybe I am just being pedantic.
---
### Author:  [@alejandromerchan](https://discourse.julialang.org/u/alejandromerchan)
#### Post date: [June 24, 2020, 7:31pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/16 "2020-06-24T19:31:37Z")
I like that one of the comments, if my German is correct, says that he doesn’t care about comparisons to R and Python, and want to see comparisons between Julia and FORTRAN. I wonder how many data analyst really use FORTRAN knowingly and on purpose?
---
### Author:  [@viralbshah](https://discourse.julialang.org/u/viralbshah)
#### Post date: [June 24, 2020, 7:45pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/17 "2020-06-24T19:45:21Z")
> [@stevengj](#):
>
> I can’t help thinking that this framing undersells Julia — the really story here is that _pure Julia_ CSV reading beats _highly optimized C libraries_ that are popular to call from Python and R.
>
> (Though this is explained in the second paragraph.)
This is very true. I wish we could talk up the fact that you can do this in Julia all the way. In my experience, people are never convinced with that argument as much as they are convinced by benchmark numbers.
I always @jeff.bezanson’s line from a while ago sums it up: Come for the performance, stay for the experience.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [June 24, 2020, 8:23pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/18 "2020-06-24T20:23:08Z")
People are always moving the bar. Of course it’s a silly comment since the R and Python CSV parsers are written in C and there’s no reason Fortran would be faster. At this point, it’s entirely possible that CSV.jl is the fastest overall CSV parser in existence.
---
### Author:  [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)
#### Post date: [June 24, 2020, 9:22pm UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/19 "2020-06-24T21:22:00Z")
This is a great first step. Hopefully dataframes will be as fast as data.table in the future.
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [June 25, 2020, 4:12am UTC](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974/20 "2020-06-25T04:12:46Z")
I think the state-of-the-art CSV reader on Python these days is pyarrow, not pandas. It also is way faster than fread from R. I believe the team that originally created pandas has long ago moved all their efforts over to the arrow/pyarrow project. I’ve been running a fairly comprehensive CSV benchmarking comparison for a couple of years [here](https://www.queryverse.org/benchmarks/), and the parallel version of pyarrow is the thing to beat these days
[Next page](https://discourse.julialang.org/t/csv-reader-benchmarks-julia-reads-csvs-10-20x-faster-than-python-and-r/41974.md?page=2)