# Improving serialization time

**URL:** <https://discourse.julialang.org/t/improving-serialization-time/11318>\
**Category:** Performance\
**Created:** [June 1, 2018, 2:47am UTC](https://discourse.julialang.org/t/improving-serialization-time/11318 "2018-06-01T02:47:23Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![jinliangwei](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jinliangwei/32/1929_2.png) [@jinliangwei](https://discourse.julialang.org/u/jinliangwei)\
**Post date:** [June 1, 2018, 2:47am UTC](https://discourse.julialang.org/t/improving-serialization-time/11318/1 "2018-06-01T02:47:23Z")

</div>

I have a object which is essentially `a = [[(Int64(10), Int32(2)) for i = 1:500] for i = 1:102661]`. Serializing it takes 1.9 seconds. This has become a major bottleneck in my program because it has to be down repeatedly. Is there anything I can do to reduce the time spent on serialization?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [June 1, 2018, 2:51am UTC](https://discourse.julialang.org/t/improving-serialization-time/11318/2 "2018-06-01T02:51:23Z")

</div>

Can you post a MWE?

Also more details.  
Are you serializing with JLD? JLD2?  
`Base.serialize`  
something else?

---

<div class="post-metadata">

**Author:** ![jinliangwei](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jinliangwei/32/1929_2.png) [@jinliangwei](https://discourse.julialang.org/u/jinliangwei)\
**Post date:** [June 1, 2018, 5:48am UTC](https://discourse.julialang.org/t/improving-serialization-time/11318/3 "2018-06-01T05:48:55Z")

</div>

```julia
julia> a = [[(Int64(10), Int32(2)) for i = 1:500] for i = 1:102661]
julia> buffer = IOBuffer()
julia> @time serialize(buffer, a)
  1.482290 seconds (395.62 k allocations: 1.012 GiB, 5.20% gc time)

```

My apologies for the incomplete information. Here is an MWE.

I am using `Base.serialize`.

Just a little more about the application. I have a parallel computing application written in C++ with Julia embedded. There are many processes that form a “ring” topology. Essentially each process holds a partition of that `a` matrix, applies some updates to its partition by processing its local data and then sends its `a` partition to the next process in the ring. This process has to repeat many times.

---

<div class="post-metadata">

**Author:** ![jinliangwei](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jinliangwei/32/1929_2.png) [@jinliangwei](https://discourse.julialang.org/u/jinliangwei)\
**Post date:** [June 1, 2018, 5:59am UTC](https://discourse.julialang.org/t/improving-serialization-time/11318/4 "2018-06-01T05:59:00Z")

</div>

Would I likely reduce the serialization time if I use a different serializer or different data structure to store the same data?

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [June 1, 2018, 1:31pm UTC](https://discourse.julialang.org/t/improving-serialization-time/11318/5 "2018-06-01T13:31:02Z")

</div>

> [@jinliangwei](#):
>
> Would I likely reduce the serialization time if I use a different serializer or different data structure to store the same data?

No, `Base.serialize` is generally faster than any other serializers.

Using a different Data Structure might help (If you were using a different serialize it would be easy to know if it would or not; but they would be slower than `Base.serialze` anyway)

---

<div class="post-metadata">

**Author:** ![avik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/avik/32/17_2.png) [@avik](https://discourse.julialang.org/u/avik)\
**Post date:** [June 1, 2018, 1:47pm UTC](https://discourse.julialang.org/t/improving-serialization-time/11318/6 "2018-06-01T13:47:52Z")

</div>

Are your processes on the same machine? Are you sending the data from julia memory to C?

For many julia processes on the same machine, using `SharedArrays` might be better than using serialisation.

As far as possible, you want to transfer the memory as-is. How to do that efficiently is unfortunately highly application dependent.

---

<div class="post-metadata">

**Author:** ![jinliangwei](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jinliangwei/32/1929_2.png) [@jinliangwei](https://discourse.julialang.org/u/jinliangwei)\
**Post date:** [June 1, 2018, 1:57pm UTC](https://discourse.julialang.org/t/improving-serialization-time/11318/7 "2018-06-01T13:57:44Z")

</div>

Thanks! Not all are on the same machine but many are. My understanding is that in one process I may have only one thread that calls Julia functions. Otherwise, I could just use shared memory among my workers.

---

<div class="post-metadata">

**Author:** ![Juser](https://avatars.discourse-cdn.com/v4/letter/j/34f0e0/32.png) [@Juser](https://discourse.julialang.org/u/Juser)\
**Post date:** [June 2, 2018, 4:27am UTC](https://discourse.julialang.org/t/improving-serialization-time/11318/8 "2018-06-02T04:27:58Z")

</div>

> Blockquote My understanding is that in one process I may have only one thread that calls Julia functions. Otherwise, I could just use shared memory among my workers.

I could be misunderstanding what you mean, but I thought Julia had [multi-threading](https://docs.julialang.org/en/latest/base/multi-threading/).