# Serialize or swap file?

**URL:** <https://discourse.julialang.org/t/serialize-or-swap-file/44242>\
**Category:** New to Julia\
**Tags:** question\
**Created:** [August 4, 2020, 9:49am UTC](https://discourse.julialang.org/t/serialize-or-swap-file/44242 "2020-08-04T09:49:34Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![andrey2185](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andrey2185/32/9889_2.png) [@andrey2185](https://discourse.julialang.org/u/andrey2185)\
**Post date:** [August 4, 2020, 9:49am UTC](https://discourse.julialang.org/t/serialize-or-swap-file/44242/1 "2020-08-04T09:49:34Z")

</div>

Hello!  
When processing data that does not fit in RAM, what is more efficient - manual serialization or  
swap file?

Splitting the algorithm into subtasks will not work, because at each stage of the calculations, any data from the available data may be needed

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [August 4, 2020, 10:05am UTC](https://discourse.julialang.org/t/serialize-or-swap-file/44242/2 "2020-08-04T10:05:15Z")

</div>

Have you thought about  
[https://docs.julialang.org/en/v1/stdlib/Mmap/](https://docs.julialang.org/en/v1/stdlib/Mmap/)  
?

---

<div class="post-metadata">

**Author:** ![andrey2185](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andrey2185/32/9889_2.png) [@andrey2185](https://discourse.julialang.org/u/andrey2185)\
**Post date:** [August 4, 2020, 10:33am UTC](https://discourse.julialang.org/t/serialize-or-swap-file/44242/3 "2020-08-04T10:33:21Z")

</div>

thanks,

but I don’t want to write queries to a giant file or rewrite it when needed

it is better and clearer to have many typical files and two functions: serialize (), deserialize ()

but I would like not to think about this too)

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [August 4, 2020, 10:35am UTC](https://discourse.julialang.org/t/serialize-or-swap-file/44242/4 "2020-08-04T10:35:33Z")

</div>

Please have a look at [GitHub - xiaodaigh/JDF.jl: Julia DataFrames serialization format](https://github.com/xiaodaigh/JDF.jl)

On the hardware side of things, if you need more capacity consider ZRAM on Linux, which is using RAM as a compressed swap file

> **[How to enable the zRAM module for faster swapping on Linux](https://www.techrepublic.com/article/how-to-enable-the-zram-module-for-faster-swapping-on-linux/)**
>
> If you're finding your Linux system performance not quite up to par, enable zRAM for a more efficient swap system.

Also if you have the option of new hardware there is Intel Optane memory which acts as a slower, cheaper tier of memory

---

<div class="post-metadata">

**Author:** ![andrey2185](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andrey2185/32/9889_2.png) [@andrey2185](https://discourse.julialang.org/u/andrey2185)\
**Post date:** [August 4, 2020, 6:57pm UTC](https://discourse.julialang.org/t/serialize-or-swap-file/44242/5 "2020-08-04T18:57:37Z")

</div>

very cool, when deserializing a 5x1000000 Int32 jdf dataframe, the speed is 460 times faster than deserializing out of the box, how is this possible?