# Storing data in Julia

**URL:** <https://discourse.julialang.org/t/storing-data-in-julia/21854>\
**Category:** New to Julia\
**Created:** [March 14, 2019, 8:40am UTC](https://discourse.julialang.org/t/storing-data-in-julia/21854 "2019-03-14T08:40:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ennvvy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ennvvy/32/7816_2.png) [@ennvvy](https://discourse.julialang.org/u/ennvvy)\
**Post date:** [March 14, 2019, 8:40am UTC](https://discourse.julialang.org/t/storing-data-in-julia/21854/1 "2019-03-14T08:40:07Z")

</div>

I am trying to save the outcome of a Monte Carlo simulation in Julia - 100 MC runs with 3 100-by-100 matrices, 2 100-by-1 arrays and 100 directed graphs. However, the size of the data comes up to 9gb. I currently use save(“filename.jld”). Is there a way to compress and store?

---

<div class="post-metadata">

**Author:** ![mauro3](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mauro3/32/292_2.png) [@mauro3](https://discourse.julialang.org/u/mauro3)\
**Post date:** [March 14, 2019, 8:47am UTC](https://discourse.julialang.org/t/storing-data-in-julia/21854/2 "2019-03-14T08:47:00Z")

</div>

There seems to be a `compress=true` option. Have you tried it? Also, maybe the graphs can be stored more efficiently by converting it to a sparse array? Or maybe use something in [https://github.com/JuliaGraphs/GraphIO.jl](https://github.com/JuliaGraphs/GraphIO.jl)?

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [March 14, 2019, 2:34pm UTC](https://discourse.julialang.org/t/storing-data-in-julia/21854/3 "2019-03-14T14:34:20Z")

</div>

Compression could help a lot if you have repeated data, but unless disk space is your only concern, I’d first look into if you can reduce what you’re storing. Even if it compresses very efficiently, it’ll probably be slow to compress and decompress the data. And 9 GB puts a lot of pressure on your memory.

Assume 100x100 nodes, and 100 runs, the amount you’re storing per node pair is `9*2^30/(100*100^2) ≈ 9664` bytes. What is taking up all this space? What do the structures look like that you’re storing?

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [March 14, 2019, 2:56pm UTC](https://discourse.julialang.org/t/storing-data-in-julia/21854/4 "2019-03-14T14:56:01Z")

</div>

One thing that’s good to check is which of your matrices, if any, can be sparse. Julia has [pretty good](https://docs.julialang.org/en/v1/stdlib/SparseArrays/) sparse arrays in the stdlib. Even if you don’t want to actually _use_ sparse matrices, you could make them sparse just for loading and storing. If you are storing your graphs as matrices, I’m guessing they should probably be sparse. From your description it sounds like your issue is with the graphs somehow.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 14, 2019, 7:51pm UTC](https://discourse.julialang.org/t/storing-data-in-julia/21854/5 "2019-03-14T19:51:58Z")

</div>

> [@ExpandingMan](#):
>
> If you are storing your graphs as matrices, I’m guessing they should probably be sparse.

(In that case they should also compress extremely well.)