# Recommended serialization for distributed file systems (and is old jld2 data gone forever)?

**URL:** <https://discourse.julialang.org/t/recommended-serialization-for-distributed-file-systems-and-is-old-jld2-data-gone-forever/65759>\
**Category:** General Usage\
**Tags:** question, jld2, distributed, serialization\
**Created:** [August 3, 2021, 7:56pm UTC](https://discourse.julialang.org/t/recommended-serialization-for-distributed-file-systems-and-is-old-jld2-data-gone-forever/65759 "2021-08-03T19:56:50Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mkarikom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkarikom/32/7096_2.png) [@mkarikom](https://discourse.julialang.org/u/mkarikom)\
**Post date:** [August 3, 2021, 7:56pm UTC](https://discourse.julialang.org/t/recommended-serialization-for-distributed-file-systems-and-is-old-jld2-data-gone-forever/65759/1 "2021-08-03T19:56:50Z")

</div>

I’m trying to read .jld2 files from a cluster dfs share and keep getting stuff like:

```julia
BoundsError: attempt to access 23-element Vector{Int64} at index [31]
Stacktrace:
  [1] getindex
    @ ./array.jl:801 [inlined]
  [2] getindex
    @ ./abstractarray.jl:1173 [inlined]
  [3] read_heap_object(f::JLD2.JLDFile{JLD2.MmapIO}, hid::JLD2.GlobalHeapID, rr::JLD2.ReadRepresentation{JLD2.RelOffset, JLD2.RelOffset})
    @ JLD2 ~/.julia/packages/JLD2/sFiXk/src/global_heaps.jl:130
  [4] jlconvert
    @ ~/.julia/packages/JLD2/sFiXk/src/data/writing_datatypes.jl:327 [inlined]
  [5] types_from_refs(f::JLD2.JLDFile{JLD2.MmapIO}, ptr::Ptr{Nothing})
    @ JLD2 ~/.julia/packages/JLD2/sFiXk/src/data/reconstructing_datatypes.jl:290
  [6] jlconvert(rr::JLD2.ReadRepresentation{DataType, JLD2.OnDiskRepresentation{(0, 16), Tuple{String, Vector{Any}}, Tuple{JLD2.Vlen{String}, JLD2.Vlen{JLD2.RelOffset}}}()}, f::JLD2.JLDFile{JLD2.MmapIO}, ptr::Ptr{Nothing}, header_offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/sFiXk/src/data/reconstructing_datatypes.jl:311

```

There is a closed (pre/post the jld2 dev hiatus?) issue on github where [a workaround for writing was suggested](https://github.com/JuliaIO/JLD2.jl/issues/55#issuecomment-366166343), but the following adaptation (I really have no idea what I’m doing here) fails:

```julia
jldopen(fn, true, true, true, IOStream) do file
  for (key, value) in dictionary
    check2 = read(file, key, value)
  end
end
ERROR: UndefVarError: dictionary not defined
Stacktrace:
 [1] (::var"#15#16")(file::JLD2.JLDFile{IOStream})
   @ Main ./REPL[33]:2
 [2] jldopen(::var"#15#16", ::String, ::Vararg{Any, N} where N; kws::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ JLD2 ~/.julia/packages/JLD2/sFiXk/src/loadsave.jl:4
 [3] jldopen(::Function, ::String, ::Bool, ::Bool, ::Bool, ::Type)
   @ JLD2 ~/.julia/packages/JLD2/sFiXk/src/loadsave.jl:2
 [4] top-level scope

```

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [August 31, 2021, 3:09pm UTC](https://discourse.julialang.org/t/recommended-serialization-for-distributed-file-systems-and-is-old-jld2-data-gone-forever/65759/2 "2021-08-31T15:09:14Z")

</div>

Hi @mkarikom,

If this is an old file, prior to the issue #55 (the one you linked) being fixed,  
then - sadly - there is nothing you can do. There was an error during writing  
and the data never got written which is only noticed upon reading.  
The error message is a bit odd, though.

The workaround you posted was meant to avoid the problem while writing files. It can’t help you during reading.