Some of you might have noticed that TextParse.jl (and CSVFiles.jl , which is a small wrapper around TextParse.jl ) saw some major performance regressions initially on julia 1.0. I just fixed these and now both packages are back to the kind of performance that we saw on julia 0.6 for them (which was p…

A side note is that fread is very fast at most everything that it does!

Thanks a lot for your effort and these extensive benchmarks. I was wondering about the number of columns per data set, it would be great to see how each approach scales as a function of that. Last time I checked, most julia solutions (except DataFrame’s deprecated readtable and Base’s readdlm) were …

I have asked before but I wonder whether Julia van eventually match fread.

There is no theoretical reason it couldn’t, someone just needs to put in the micro-optimization work. I think that fixing major performance regressions is important, so I am happy that TextParse.jl is competitive again, but I am not sure that it is super-important to beat data.table::fread once we …

[image] Tamas_Papp: large files (when this matters the most) And to add to that, TextParse.jl looks already quite comparable to fread for larger numbers of rows. If one has to read many small csv files (which is also an important use case), TextParse.jl may be slower for now, but coincidental…

Looks great! Out of curiosity, can you expand a bit on why you perform better than CSV for large datasets?

[image] jtackm: I was wondering about the number of columns per data set, it would be great to see how each approach scales as a function of that. Agreed, a PR that adds runs with say 100, 1k and 10k columns would be great! [image] Tamas_Papp: I think that fixing major performance regres…

[image] davidanthoff: Agreed, a PR that adds runs with say 100, 1k and 10k columns would be great! Sure, I’ll have a look

Thanks to a PR from @jtackm we now also have results for files with 200 columns: [cols_200_withoutna] direct link to figure [cols_200_withna] direct link to figure

Do you have any ideas as to why there seems to be substantial overhead between CSVFiles and TextParse? Sometimes the time difference looks negligible, but typically I see CSVFiles lagging behind, even though it is TextParse under the hood.

TextParse.jl is fast again

Specific Domains Data

Topic		Replies	Views
CSV parsing performance Data	4	1460	July 4, 2017
CSV read performance vs Pandas General Usage	29	8591	May 6, 2019
[ANN] TableReader.jl - A fast and simple CSV parser Package Announcements package , announcement , data , csv	24	6218	March 28, 2019
CSV Reading (rewrite in C?) Internals & Design	50	5752	October 1, 2018
CSV read in is too slow than other language General Usage performance	13	1692	June 21, 2023

TextParse.jl is fast again

Related topics