Hi,
is there any way to read a 88 GB csv file into a DataFrame? It kills julia, even with the -t auto option.
Thanks!
How much RAM does your machine have ie Sys.total_physical_memory()/(2^30) # GiB ?
If you don’t need to materialize the entire dataset in memory, consider tyring with DuckDB—either directly using DuckDB or using the excellent frontend QuackIO.jl. DuckDB allows you to load the data lazily, materializing only the subset of the data you need.
62.449…
Definitely not enough RAM for typical in-memory table libraries to completely read a file of that size. Look into chrisnekarda’s suggestion.
I’ll add that the size of the CSV in memory depends on variable types.
For example, you might have a column of strings, but there are only 1000 unique strings. If these strings are long, (say 20 characters), they are going to take up an enormous amount of memory stored as strings. In contrast, if you store them as categorical arrays, then you have a length 1000 String array (which takes up approximately zero memory) and you have a huge possibly UInt16 refs array (default is UInt32 for CategoricalArrays but you can compress to UInt16). So moving from strings to UINTs reduces your memory footprint by a factor of 80 in this example.
Of course this does not solve your immediate problem – if you can change variable types to reduce memory only after loading the data with CSV.read, then you still run out of memory before loading the data.
One thing I have done is process a large number of smaller CSV files into strongly typed HDF5 files, then join the columns together to form a larger HDF5 file. This larger HDF5 gives me a smaller memory footprint (and also allows me to rapidly load the specific variables I need for an analysis).
CSV was not designed for data of this size.
Allowing an 88 GB CSV file to exist is environmentally unfriendly.
Julia DatafFrames need to fit into memory, therefore are not very suitable for this amount of data. Available memory could be enlarged by using swap space, and this would require full control over the system, reboot, details depend on the system (Linux, Windows). The mmap function maps memory into file space. Afaik, Julia DataFrames do not support mmap. The csv file itself could be mmaped. With an index array to the byte offsets of each line, this would be something like a “poor man’s data frame.”
SQLite can convert csv to a database table. This would not change things fundamentally, instead of millions of lines, there are then millions of rows. But SQL has tools and window functions which make working with such data more convenient. These would be available via the SQLite.jl package.
Another possibility is to convert the CSV file to Arrow format, perhaps through DuckDB or Polars. The Arrow package for Julia, which presents a columntable interface to an Arrow file, memory-maps the file and is able to process files larger than physical memory.
Arrow also supports categorical data in what they call dict-encoded columns. Often CSV files are very large because they have thousands or millions of repetitions of the same strings in a particular column.
I think the individual columns of the dataframe can be AbstractVectors that are saved on disk and “loaded” via mmap, so DataFrames can work for larger than RAM data. Saving the data in the .arrow format (as suggested above) and loading with Arrow.Table, either as Arrow.Table(file) or DataFrame(Arrow.Table(file)), gives you a memory mapped table in Julia. I might personally still prefer the plain Arrow.Table(file), as I’m not sure how many DataFrames operations are optimized for memory mapped columns, whereas the Arrow.Table is clearly designed for this use case.
You could also try Arrow.Stream(file) (see here) and then get an iterator of tables via Tables.partitions (Arrow files are subdivided into many smaller tables that can be streamed one by one, see the .arrows stream format).
To convert from .csv to .arrow (or actually .arrows, the streaming format), DuckDB is a great option, but you can do it also in julia (see User Manual · Arrow.jl) though I haven’t tested this latter option on large datasets.