Hi,
is there any way to read a 88 GB csv file into a DataFrame? It kills julia, even with the -t auto option.
Thanks!
How much RAM does your machine have ie Sys.total_physical_memory()/(2^30) # GiB ?
If you don’t need to materialize the entire dataset in memory, consider tyring with DuckDB—either directly using DuckDB or using the excellent frontend QuackIO.jl. DuckDB allows you to load the data lazily, materializing only the subset of the data you need.
62.449…
Definitely not enough RAM to handle a file of that size for typical in-memory table libraries on typical operating systems of that size. Look into chrisnekarda’s suggestion.
I’ll add that the size of the CSV in memory depends on variable types.
For example, you might have a column of strings, but there are only 1000 unique strings. If these strings are long, (say 20 characters), they are going to take up an enormous amount of memory stored as strings. In contrast, if you store them as categorical arrays, then you have a length 1000 String array (which takes up approximately zero memory) and you have a huge possibly UInt16 refs array (default is UInt32 for CategoricalArrays but you can compress to UInt16). So moving from strings to UINTs reduces your memory footprint by a factor of 80 in this example.
Of course this does not solve your immediate problem – if you can change variable types to reduce memory only after loading the data with CSV.read, then you still run out of memory before loading the data.
One thing I have done is process a large number of smaller CSV files into strongly typed HDF5 files, then join the columns together to form a larger HDF5 file. This larger HDF5 gives me a smaller memory footprint (and also allows me to rapidly load the specific variables I need for an analysis).
CSV was not designed for data of this size.
Allowing an 88 GB CSV file to exist is environmentally unfriendly.