JLD2 MethodError when dealing with large dictionaries?

New to Julia
1

I have a numeric analysis code currently running on a cluster which serialises a dictionary contains several multi-million point arrays using the JLD2 library. The file is saved using @save filename source and read back using @load filename source. In previous runs using smaller arrays this has worked fine, however it has begun throwing a lengthy MethodError with these larger arrays, the first few lines of which are as follows:

ERROR: MethodError: no method matching read_array(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::JLD2.FixedLengthString{String}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1})
Closest candidates are:
  read_array(::JLD2.JLDFile, ::JLD2.ReadDataspace, ::JLD2.ReadRepresentation{T,RR}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Union{Nothing, Array{JLD2.ReadAttribute,1}}) where {T, RR} at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:312
Stacktrace:
 [1] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.FixedLengthString{String}, ::Array{JLD2.ReadAttribute,1}) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:173
 [2] macro expansion at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:156 [inlined]
 [3] macro expansion at /home/dave/.julia/packages/JLD2/I2BGy/src/datatypes.jl:76 [inlined]
 [4] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:152
 [5] load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:92
 [6] jlconvert at /home/dave/.julia/packages/JLD2/I2BGy/src/data.jl:674 [inlined]
 [7] macro expansion at /home/dave/.julia/packages/JLD2/I2BGy/src/data.jl:1325 [inlined]

I have been struggling to resolve this issue for several days now and cannot find anything online which explains this behaviour. The current file is only 1GB but does JLD2 have a limit on file size? Is this just a limitation of the @save/@load macros? Any help would be greatly appreciated!

Version details:

$ julia -v
julia version 1.3.1

julia> Pkg.installed()
  "JLD2"             => v"0.1.11"
1 Like
2

Update:

I updated my julia installation and reinstalled JLD2, but the issue persists. I am able to recreate the full error with the following:

$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.0 (2020-03-21)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> using JLD2

julia> @load "filename.jld2" source
ERROR: MethodError: no method matching read_array(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::JLD2.FixedLengthString{String}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1})
Closest candidates are:
  read_array(::JLD2.JLDFile, ::JLD2.ReadDataspace, ::JLD2.ReadRepresentation{T,RR}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Union{Nothing, Array{JLD2.ReadAttribute,1}}) where {T, RR} at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:312
Stacktrace:
 [1] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.FixedLengthString{String}, ::Array{JLD2.ReadAttribute,1}) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:173
 [2] macro expansion at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:156 [inlined]
 [3] macro expansion at /home/dave/.julia/packages/JLD2/I2BGy/src/datatypes.jl:76 [inlined]
 [4] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:152
 [5] load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:92
 [6] jlconvert at /home/dave/.julia/packages/JLD2/I2BGy/src/data.jl:674 [inlined]
 [7] macro expansion at /home/dave/.julia/packages/JLD2/I2BGy/src/data.jl:1325 [inlined]
 [8] jlconvert(::JLD2.ReadRepresentation{Pair{String,Any},JLD2.OnDiskRepresentation{(0, 16),Tuple{String,Any},Tuple{JLD2.Vlen{String},JLD2.RelOffset}}()}, ::JLD2.JLDFile{JLD2.MmapIO}, ::Ptr{Nothing}, ::JLD2.RelOffset) at /home/dave/.julia/packages/JLD2/I2BGy/src/data.jl:1272
 [9] read_scalar at /home/dave/.julia/packages/JLD2/I2BGy/src/dataio.jl:37 [inlined]
 [10] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{Pair{String,Any},JLD2.OnDiskRepresentation{(0, 16),Tuple{String,Any},Tuple{JLD2.Vlen{String},JLD2.RelOffset}}()}, ::Array{JLD2.ReadAttribute,1}) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:171
 [11] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:149
 [12] load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:92
 [13] jlconvert at /home/dave/.julia/packages/JLD2/I2BGy/src/data.jl:674 [inlined]
 [14] macro expansion at /home/dave/.julia/packages/JLD2/I2BGy/src/dataio.jl:70 [inlined]
 [15] macro expansion at ./simdloop.jl:77 [inlined]
 [16] read_array!(::Array{Pair{String,Any},1}, ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{Pair{String,Any},JLD2.RelOffset}) at /home/dave/.julia/packages/JLD2/I2BGy/src/dataio.jl:68
 [17] read_array(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::JLD2.ReadRepresentation{Pair{String,Any},JLD2.RelOffset}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:323
 [18] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{Any,JLD2.RelOffset}, ::Array{JLD2.ReadAttribute,1}) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:198
 [19] macro expansion at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:156 [inlined]
 [20] macro expansion at /home/dave/.julia/packages/JLD2/I2BGy/src/datatypes.jl:76 [inlined]
 [21] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:152
 [22] load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:92
 [23] jlconvert at /home/dave/.julia/packages/JLD2/I2BGy/src/data.jl:674 [inlined]
 [24] jlconvert(::JLD2.ReadRepresentation{Dict{String,Any},JLD2.CustomSerialization{Array,JLD2.RelOffset}}, ::JLD2.JLDFile{JLD2.MmapIO}, ::Ptr{Nothing}, ::JLD2.RelOffset) at /home/dave/.julia/packages/JLD2/I2BGy/src/data.jl:576
 [25] read_scalar at /home/dave/.julia/packages/JLD2/I2BGy/src/dataio.jl:37 [inlined]
 [26] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{Dict{String,Any},JLD2.CustomSerialization{Array,JLD2.RelOffset}}, ::Array{JLD2.ReadAttribute,1}) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:171
 [27] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:149
 ... (the last 23 lines are repeated 2 more times)
 [74] load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:92
 [75] getindex(::JLD2.Group{JLD2.JLDFile{JLD2.MmapIO}}, ::String) at /home/dave/.julia/packages/JLD2/I2BGy/src/groups.jl:108
 [76] read(::JLD2.JLDFile{JLD2.MmapIO}, ::String) at /home/dave/.julia/packages/JLD2/I2BGy/src/JLD2.jl:326
 [77] #3 at /home/dave/.julia/packages/JLD2/I2BGy/src/loadsave.jl:77 [inlined]
 [78] jldopen(::var"#3#4", ::String; kws::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/dave/.julia/packages/JLD2/I2BGy/src/loadsave.jl:4
 [79] jldopen(::Function, ::String) at /home/dave/.julia/packages/JLD2/I2BGy/src/loadsave.jl:2

Updated version info:

$ julia -v
julia version 1.4.0

julia> Pkg.installed()["JLD2"]
┌ Warning: Pkg.installed() is deprecated
â”” @ Pkg /build/julia/src/julia-1.4.0/usr/share/julia/stdlib/v1.4/Pkg/src/Pkg.jl:531
v"0.1.11"
3

I have not had positive experiences with JLD and JLD2; and would recommend using JSON and HDF5. I’ve “complained” about it here:

(though I haven’t tried it again recently…)

2 Likes
4

I had come across your post in my recent searches for a solution, along with numerous others including open issues on the GitHub dating back years. However, I had (perhaps naĂŻvely) assumed that those issues would have been resolved by this point and that the dev(s) had simply forgotten to address/close them.

Thanks for your suggestions, I guess I will need to quickly cobble something together. I just hope others don’t make the same mistake I did and find this issue in the final month of their PhDs while running large set numeric analyses!

5

I am getting the same error when trying to read JLD2 files saved last week:

julia> @load "D:\\model runs\\savedruns_apr5.jld2"
ERROR: MethodError: no method matching read_array(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::JLD2.FixedLengthString{String}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1})
Closest candidates are:
  read_array(::JLD2.JLDFile, ::JLD2.ReadDataspace, ::JLD2.ReadRepresentation{T,RR}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Union{Nothing, Array{JLD2.ReadAttribute,1}}) where {T, RR} at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:312
Stacktrace:
 [1] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.FixedLengthString{String}, ::Array{JLD2.ReadAttribute,1}) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:173
 [2] macro expansion at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:156 [inlined]
 [3] macro expansion at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datatypes.jl:76 [inlined]
 [4] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:152
 [5] load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:92
 [6] jlconvert at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\data.jl:674 [inlined]
 [7] macro expansion at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\data.jl:1290 [inlined]
 [8] jlconvert at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\data.jl:1237 [inlined]
 [9] read_scalar at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\dataio.jl:37 [inlined]
 [10] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{Pair{Any,Any},JLD2.OnDiskRepresentation{(0, 8),Tuple{Any,Any},Tuple{JLD2.RelOffset,JLD2.RelOffset}}()}, ::Array{JLD2.ReadAttribute,1}) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:171
 [11] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:149
 [12] load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:92
 [13] jlconvert at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\data.jl:674 [inlined]
 [14] macro expansion at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\dataio.jl:70 [inlined]
 [15] macro expansion at .\simdloop.jl:77 [inlined]
 [16] read_array!(::Array{Pair{Any,Any},1}, ::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{Pair{Any,Any},JLD2.RelOffset}) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\dataio.jl:68
 [17] read_array(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::JLD2.ReadRepresentation{Pair{Any,Any},JLD2.RelOffset}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:323
 [18] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{Any,JLD2.RelOffset}, ::Array{JLD2.ReadAttribute,1}) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:198
 [19] macro expansion at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:156 [inlined]
 [20] macro expansion at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datatypes.jl:76 [inlined]
 [21] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:152
 [22] load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:92
 [23] jlconvert at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\data.jl:674 [inlined]
 [24] jlconvert(::JLD2.ReadRepresentation{Dict{Any,Any},JLD2.CustomSerialization{Array,JLD2.RelOffset}}, ::JLD2.JLDFile{JLD2.MmapIO}, ::Ptr{Nothing}, ::JLD2.RelOffset) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\data.jl:576
 [25] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadRepresentation{Dict{Any,Any},JLD2.CustomSerialization{Array,JLD2.RelOffset}}, ::Array{JLD2.ReadAttribute,1}) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\dataio.jl:37
 [26] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:149
 [27] load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\datasets.jl:92
 [28] getindex(::JLD2.Group{JLD2.JLDFile{JLD2.MmapIO}}, ::String) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\groups.jl:108
 [29] read(::JLD2.JLDFile{JLD2.MmapIO}, ::String) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\JLD2.jl:326
 [30] #7 at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\loadsave.jl:77 [inlined]
 [31] #jldopen#33(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(jldopen), ::var"#7#8", ::String) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\loadsave.jl:4
 [32] jldopen(::Function, ::String) at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\loadsave.jl:2
 [33] top-level scope at C:\Users\niclas\.julia\packages\JLD2\w2vgv\src\loadsave.jl:76
 [34] eval(::Module, ::Any) at .\boot.jl:330
 [35] eval_user_input(::Any, ::REPL.REPLBackend) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\REPL\src\REPL.jl:86
 [36] run_backend(::REPL.REPLBackend) at C:\Users\niclas\.julia\packages\Revise\Pcs5V\src\Revise.jl:1073
 [37] top-level scope at none:0

One JLD2 file is extremely large (30 GB) and contains customs structs, but the same error occurs with a small file (150 kB) that only holds two vanilla Julia dicts. I don’t think any Julia packages have updated after writing the files, but I did notice that my computer had restarted even though I left it on - so it is likely Windows Update did something. Could that be the culprit?

The runs saved in these files were the result of running a model for 8 days straight. Please don’t tell me that I lost all this work and can’t recover it.

EDIT: The problem persists when I use the alternate syntax jldopen(filename, "r"). I get a listing of the variables in the file, but when I try to read the actual data I get the same error as above.

EDIT2: Added a screenshot of the beginning of the error message to highlight the main differences in method signatures, namely ::JLD2.ReadRepresentation{T,RR} vs ::JLD2.FixedLengthString{String}.

repl_snip
repl_snip1183Ă—128 8.94 KB

6

Hi Niclas,

I am using Linux, so if you are experiencing the same issue on Windows then I can say definitively that the error is not OS-related.

I will be looking into this issue further once I have submitted my final thesis at the end of this month, but for the moment my own conclusion is that it is not the total data size, but instead the array sizes that comprise the data. For example, one of my test cases uses arrays of size 64x64x64x1 and that data reads/write fine. The next test case uses arrays of size 128x128x128x1 and JLD2 throws the error.

Unfortunately, I have found no way of recovering the data once JLD2 has touched it. I have learned my lesson and sworn off JLD2, instead replacing it with BSON, which supports the data structures I am employing – seemingly without drawback (even the file sizes are slightly smaller). If you are interested in something quick and dirty but functional, I am using the following code in lieu of JLD2:

 using BSON
 
 function write(source::Dict{String, Any}, filename::String = "output.bson")
     println("Writing file $filename...")
     flush(stdout)
     bson(filename, source)
 end #function write
 
 function read(filename::String = "output.bson")
     println("Reading file $filename...")
     flush(stdout)
     source = BSON.load(filename)
     return source
 end #function read
7

I believe there have now been several threads recommending to stay away from JLD and JLD2. For myself I’ve switched to JSON and HDF5