Help with Arrow.jl and size of files

Specific Domains Data
,
21

Thank you, I’ve always struggled with conversions in Julia. I will use these from now on.

22

Thanks for your comments. I’ll probably stick to Arrow then.

23

Sorry, forgot to reply, no I didn’t catch that at first, but now I see it, thank you. I’ll try that after Int8 and see if I can get further reduction and what it does to speed on both writing and reading.

1 Like
24

Okay, with conversion to Int8 with your help and with compression using compress = :lz4 I got it down to 93Mb from the original 3.4Gb. Thanks to everyone for the help! Both with and without compression was much faster than CSV (with spaces). 205 sec vs 37 sec (base Arrow) and 43 sec (with compression).

5 Likes