Help with Arrow.jl and size of files

Specific Domains Data
,
3

First DataFrame(my_array) normally should error, and instead DataFrame(my_array, :auto) would be needed. What is the type of my_array?

Now a crucial thing is that you should make your data to have Int8 element type. If you use Int64 this wastes a lot of memory. If you do it (I did it for data of your size):

  • CSV.jl has almost 870MB, and is slow to write;
  • Arrow.jl has 450MB, and is faster to write;

(this is without compression)

Also using Int8 will put much less burden on RAM. If instead you wanted to represent your data a Bool, then using BitArray would give you another 8x savings in RAM.

5 Likes