My crsp daily stock return data is a 6.4GB csv file. Compressed into csv.gz, it is about 1.7GB. (I posted a sample with my juliadb post, https://discourse.julialang.org/t/juliadb-getting-started-with-save-error/17891/4. ) Local macos iMac Pro 2018. Fast SSD. On R, reading the .csv.gz file into…

Conversion to DataFrame from the object that readsas returns is very fast and efficient. Less than half a second for that particular table.

To be clear, I don’t have a comparison handy at the moment. The 30 seconds on my machine could be much slower on yours, although I doubt it’d be that much slower. (I did just get a nice new NVMe SSD :grin:). In general I’ve seen good performance from SASLib and that makes me happy because it makes …

Isn’t SASlib.jl a read-only package? (It would also be pretty annoying for julia to have to resort to ancient sas for its main format…but I guess it is better than no fast storage format.)

If loading speed is a concern, CSV is probably suboptimal, Feather should be much faster to load, have you tried using it?

yes, csv is suboptimal. I had hoped that serialization would have been, given some past tests. alas, piever you are completely correct. Feather is the way to go: julia> @time x=Feather.read("crspdaily-clean.feather") 1.500487 seconds (4.97 M allocations: 251.940 MiB, 5.56% gc time) 88915607×12…

It’s quite unexpected that deserialization is slower than CSV… Can you file an issue against Julia with a minimal reproducer? Ideally you’d provide a command to generate a data frame with random values. Or even better, if you can reproduce the problem without DataFrames, just serializing/deserializi…

hi milan—I am happy to help, but this file is pretty large and experimentation is slow. Plus, the data is proprietary, so it is difficult to convey. If someone from julia computing wants to try it on my machine (it sits on the internet), I can arrange access. regards, /iaw

The timing from Feather.read is a bit misleading, though, because it actually doesn’t read any of the data from disc. All it does is read the metadata from disc (i.e. what columns are there in the file, what types do they have), and then it creates a DataFrame that will load things from disc on-dema…

@iwelch maybe you can also try this? 6G isn’t too big and I am assuming that you are reading everything into memory. [image] GitHub - tk3369/CSVReader.jl: Simple CSV reader that performs well Simple CSV reader that performs well. Contribute to tk3369/CSVReader.jl development by creatin…

Reading Data Is Still Too Slow

Specific Domains Data

xiaodai November 23, 2018, 9:35pm 14

My research back then still seem somewhat relevant Benchmarking ways to write/load DataFrames IndexedTables to disk

Topic		Replies	Views
My experiences reading CSVs from the Fannie Mae datasets Data performance , csv	62	6520	August 26, 2019
CSV read in is too slow than other language General Usage performance	13	1567	June 21, 2023
Benchmarking ways to write/load DataFrames IndexedTables to disk Data	42	7183	October 25, 2018
CSV Reading (rewrite in C?) Internals & Design	50	5416	October 1, 2018
CSV Reader Benchmarks: Julia Reads CSVs 10-20x Faster than Python and R General Usage announcement	68	9548	March 23, 2022

Reading Data Is Still Too Slow

Related topics