# Saving julia dataframes efficiently (in terms of size on the disk)
**URL:** https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151
**Category:** General Usage
**Created:** [December 11, 2019, 2:30pm UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151 "2019-12-11T14:30:58Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@curious](https://discourse.julialang.org/u/curious)
#### Post date: [December 11, 2019, 2:30pm UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/1 "2019-12-11T14:30:58Z")
What is a good format to store Julia DataFrames efficiently (in terms of size on the disk)? I have a dataset that is about 2.1GB as a Feather file. It is almost 1.9GB in Stata .dta format. But if I open it in R and save it in the .rds format, the size decreases to 239MB. Is there something like RDS format for Julia? I tried JuliaDB, but it does nothing to decrease the size on the disk.
Any recommendations for how the data could be stored on the disk.
Thanks.
---
### Author:  [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [December 11, 2019, 3:03pm UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/2 "2019-12-11T15:03:53Z")
You could try compressing Feather or another format with [gzip](https://github.com/JuliaIO/GZip.jl) or another general compression algorithm.
Feather supports compression via [CategoricalArrays](https://github.com/JuliaData/CategoricalArrays.jl). If you set columns of your table to be `CategoricalArray`s they will be stored in the feather file in a similar format. Of course, you will only get significant compression from this if your table happens to have lots of repeated values.
---
### Author:  [@ASF](https://discourse.julialang.org/u/ASF)
#### Post date: [December 11, 2019, 4:45pm UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/3 "2019-12-11T16:45:34Z")
I like [FstFileFormat](https://github.com/xiaodaigh/FstFileFormat.jl)
---
### Author:  [@curious](https://discourse.julialang.org/u/curious)
#### Post date: [December 11, 2019, 6:03pm UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/4 "2019-12-11T18:03:23Z")
Thanks. Will look into this. I suppose this only works if columns contain Categorical data. Is that right? In general, is R achieving the small size because of compression? Does it compress and decompress automatically?
---
### Author:  [@curious](https://discourse.julialang.org/u/curious)
#### Post date: [December 11, 2019, 6:03pm UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/5 "2019-12-11T18:03:58Z")
@ASF Thanks. Will try this out. Did not know about this one.
---
### Author:  [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [December 11, 2019, 6:08pm UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/6 "2019-12-11T18:08:31Z")
You can make a `CategoricalArray` out of anything, but it’ll only benefit you if the number of distinct values is much smaller than the number of elements in the array and the values are larger than a few bytes. It’s usually only useful for strings.
From the sizes you are seeing it’s obvious that R’s `rds` format is somehow compressed (the feather format is very simple, so it’s very unlikely that it’s going to “inflate” much) but I have no idea how. In extreme cases you can achieve that much from the `CategoricalArray` encoding, but more likely they are using some generalized compression algorithm (there’s a chance it’s gzip, since that is the most commonly used compression algorithm).
---
### Author:  [@curious](https://discourse.julialang.org/u/curious)
#### Post date: [December 12, 2019, 12:52am UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/7 "2019-12-12T00:52:46Z")
@ExpandingMan
Thanks very much for the explanation.
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [December 12, 2019, 1:02am UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/8 "2019-12-12T01:02:52Z")
I developed just this thing [https://github.com/xiaodaigh/JDF.jl](https://github.com/xiaodaigh/JDF.jl)
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [December 12, 2019, 1:03am UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/9 "2019-12-12T01:03:53Z")
It’s actually an R native format so the saving will be slow. Better use my other package [https://github.com/xiaodaigh/JDF.jl](https://github.com/xiaodaigh/JDF.jl) if u r happy to stay in Julia ecosystem
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [December 12, 2019, 6:01am UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/11 "2019-12-12T06:01:35Z")
R’s fst uses lots of compression algorithms. Rds is also compressed by default.
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [December 12, 2019, 6:05am UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/12 "2019-12-12T06:05:32Z")
> [@curious](#):
>
> t is almost 1.9GB in Stata .dta format. But if I open it in R and save it in the .rds format, the size decreases to 239MB
RDS does compression by default. Please try out [JDF.jl](https://github.com/xiaodaigh/JDF.jl). Based on one benchmark that I did, it’s much smaller than Feather if you have lots of strings. See
---
### Author:  [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)
#### Post date: [December 12, 2019, 6:59am UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/13 "2019-12-12T06:59:53Z")
very interesting package. any plan of adding native julia support to reading and writing fst format data?
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [December 12, 2019, 7:07am UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/14 "2019-12-12T07:07:16Z")
I would but you need c++. Which I Dont know any.
---
### Author:  [@curious](https://discourse.julialang.org/u/curious)
#### Post date: [January 19, 2020, 2:41am UTC](https://discourse.julialang.org/t/saving-julia-dataframes-efficiently-in-terms-of-size-on-the-disk/32151/15 "2020-01-19T02:41:46Z")
@xiaodai So sorry for not replying sooner. Thank you so much for these responses. I got busy with something else. But I look forward to trying this out.
Thanks again.