updated code with suggestions by ScottPJones The below took about 100~200 seconds but in R’s data.table’s fwrite the same is done in 2 seconds. It is quite a big discrepancy. I am curious as to what need to change before CSV.jl can be as fast? using DataFrames const N = 100_000_000; const K = 100…

Need to remember to try this. But why doesn’t CSV.write just do this if it’s so much faster?

Writecsv does seems to use a PipeBuffer(), so I probably didn’t test cleansave() well enough =) Pretty sure cleanload() is/was a bit faster than readcsv though

The code doesn’t work, this line writedlm(b, data) givs error ERROR: MethodError: no method matching start(::DataFrames.DataFrame)

Ah, i’m thinking of native arrays. It shouldn’t be too difficult to pull a dataframe into a type-Any matrix though? ##missing header function df2mat(dataf) ncols = length(dataf.columns); nrows = length(dataf.columns[1]); ret = Array{Any}(nrows, ncols); rmap = x->ismissing(x)?"":x; for ite…

And for the header, i came up with rhead = split(replace("$( keys(dataf) )", r"(Symbol)\[|\]|:", ""), ", "); ## return([rhead'; ret]); bigfile.csv (480,471 KB) - 66.1 seconds with cleansave, size (10,000,000 x 2) - typo? should’ve been 100 mil x 2 i think a datablobs prototype, using xSave_i32 …

Have you tried CSVFiles.jl? I’d be curious how it stacks up, I have never benchmarked it :slight_smile:

[image] xiaodai: Is that the same as running FileIO.jl? Yes, if you save as a CSV file with FileIO.jl, it will use CSVFiles.jl under the hood. TextParse.jl is actually not involved in that case, it only does the reading of files, I rolled the writing part of CSVFiles.jl myself. Do I read the…

I have 4 cores hyperthreaded. It’s a laptop high-end i7 CPU. If you run the benchmark do you see the fwrite speeds that I quoted?

I haven’t run the benchmarks. But fwrite makes use of your cores, so the more you have, the faster things should get. And yet, 4 cores is not that many, so it just seems really well done…

CSV.jl's CSV write seems slow

General Usage Performance

xiaodai February 11, 2018, 3:15am 20

Is that the same as running FileIO.jl? See my benchmarks here

Topic		Replies	Views
Multithreaded CSV writes Performance multithreading , csv	20	3768	April 14, 2023
CSV read in is too slow than other language General Usage performance	13	1746	June 21, 2023
CSV Reading (rewrite in C?) Internals & Design	50	5935	October 1, 2018
[ANN] CSV.jl 1.0 Release Package Announcements	9	1409	October 3, 2026
CSV : problem to write big dataframes Data csv	20	3126	May 29, 2023

CSV.jl's CSV write seems slow

Related topics