I have a dataset that contains several thousand gzipped CSV files each with a few hundred thousand lines. Using CSV.jl and CSV.File takes about 4 seconds, which is really good, but for the number of files till means ~3 hours just to read everything.
However, I really only need 4-5 out of the 152 columns, so I am wondering if ignoring the data I don’t need will speed this up. Is there a way to use CSV.jl (or other package) to only read certain columns of data? Or do I need to homebrew a solution?
Additional info: The columns are not fixed width, and null is used in place of any data with no values
Check out these select and drop Examples · CSV.jl
I don’t know about its impact on efficiency
You’ve tried this?
using CSV
source = CSV.Rows("your.csv", select=[1, 3])
df = DataFrames(source)
oooh, wow, I don’t know how I missed that in the documentation! Selecting the 5 columns knocks off about 0.75 seconds, which would be about 40 minutes when I’m ready to read it all! It also really reduces the amount of memory too.
I had not tried CSV.Rows, but it is giving me weird results compared to CSV.File. I imagine the speed is comparable though assuming that they are using the same code to do the reading.
Shameless plug: QuackIO.jl can do arbitrary column selectors or filtering with simple julian syntax.
Another alternative is to preprocess through a cut or awk commands in the CLI