I’m working with several large csv files. (Around 2GB each, million rows, thousands of columns).
In order to reduce the space on disk (and speed to sync online) they are compressed.
I was using R to work with them.
data.table’s fread let’s me read directly the compressed file with:
myDT <- fread("7z e -y -bso0 -so mycompress.7z", stringsAsFactors=F, na.strings=c("", "NA")) # and sometimes selecting columns or rows.
That executes transparently 7-zip and forwards the result to fread.
How can I do something similar with Julia?
I’m also considering using feather or hdf5 but I feel safer using csv for now, it’s easier for other people to access the files.