# Why JLD2.jl is 40x slower than Arrow.jl

**URL:** <https://discourse.julialang.org/t/why-jld2-jl-is-40x-slower-than-arrow-jl/122217>\
**Category:** Performance\
**Tags:** package, jld2, arrow\
**Created:** [November 4, 2024, 8:22am UTC](https://discourse.julialang.org/t/why-jld2-jl-is-40x-slower-than-arrow-jl/122217 "2024-11-04T08:22:20Z")\
**Posts on this page:** 5\
**Page:** 2

<div class="post-metadata">

**Author:** ![Sixzero](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sixzero/32/28737_2.png) [@Sixzero](https://discourse.julialang.org/u/Sixzero)\
**Post date:** [November 5, 2024, 7:57am UTC](https://discourse.julialang.org/t/why-jld2-jl-is-40x-slower-than-arrow-jl/122217/21 "2024-11-05T07:57:29Z")

</div>

Just checked HDF5.jl for comparison.

The results compared to JLD2:

```julia
HDF5 has faster append speed with big files (2-5x faster), slower when files are smaller (2-5x slower).
HDF5 full read is 10x slower. This is quite bad.
File size is 2x of what JLD2 generates.

```

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [November 5, 2024, 8:06am UTC](https://discourse.julialang.org/t/why-jld2-jl-is-40x-slower-than-arrow-jl/122217/22 "2024-11-05T08:06:40Z")

</div>

FYI: [Advanced Usage · Julia Data Format](https://juliaio.github.io/JLD2.jl/dev/advanced/#Groups-Appending-to-files)  
(but it probably won’t help much here)

JLD2 needs to parse some contents of a file before being able to append something. So, if you first read and then append, then you can save time by keeping the file open. (Open with `r+` right away)

```julia
f = jldopen(fn, "r+")
try
    read_data = JLD2.loadnesteddict(f)

    # append
    
finally
    close(f)
end

```

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [November 5, 2024, 8:08am UTC](https://discourse.julialang.org/t/why-jld2-jl-is-40x-slower-than-arrow-jl/122217/23 "2024-11-05T08:08:48Z")

</div>

Also, in your use-case, JLD2 and HDF5 should be compatible e.g. write files with JLD2 and read (edit?) with HDF5. (The reverse: writing with HDF5 and reading with JLD2 is also possible but not editing.)

---

<div class="post-metadata">

**Author:** ![Sixzero](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sixzero/32/28737_2.png) [@Sixzero](https://discourse.julialang.org/u/Sixzero)\
**Post date:** [November 5, 2024, 9:32am UTC](https://discourse.julialang.org/t/why-jld2-jl-is-40x-slower-than-arrow-jl/122217/24 "2024-11-05T09:32:11Z")

</div>

No, I am fine with the narrowest compatibility, caches can be regenerated anywhere anytime, just be efficient.

---

<div class="post-metadata">

**Author:** ![bernhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernhard/32/2619_2.png) [@bernhard](https://discourse.julialang.org/u/bernhard)\
**Post date:** [November 25, 2024, 9:59am UTC](https://discourse.julialang.org/t/why-jld2-jl-is-40x-slower-than-arrow-jl/122217/25 "2024-11-25T09:59:38Z")

</div>

![image](https://global.discourse-cdn.com/julialang/original/3X/1/5/151d0ce6c713def13c743d20d25ec79d4a6ebe0d.png)

this post has made me curious, so I did some tests on my rather plain data (all DataFrames).

The numbers \> 1.0 in the three rightmost columns are all in favour of Arrow.jl in my case.  
This was measured using `@elapsed`. Filesize is as reported by `stat` so the largest JLD2 file is 3.7GB.  
I note that the identifier column is intentionally partially hidden.

[Previous page](https://discourse.julialang.org/t/why-jld2-jl-is-40x-slower-than-arrow-jl/122217.md?page=1)
