JLD2 - load from IOBuffer

General Usage
, ,
1

I am not finding a method to load a JLD2 file using JLD2.load() from IOBuffer. This seems basic for saving and loading to cloud like S3 or Azure Blob storage (which is my case). I would like to avoid reading from a REST call and saving locally - just to load again using a filename. Perhaps I am missing something?

2

I am also searching for a way to load JLD2 using an IO stream and not a filename. Is there something ?

3

Feature request issue Load JLD2 from IOBuffer · Issue #346 · JuliaIO/JLD2.jl · GitHub

4

Bump this thread. I’m unsure what’s the status here, apparently it would be supported right now, but I’m trying to do it, and it fails.

julia> load_object(open("/home/user/test.jld2"))
ERROR: type IOStream has no field readable

In my case, what I really want is to load from a xz compressed file, via a XzDecompressorStream. I don’t care so much if the file will be completely uncompressed to memory first, not touching the disk again is great for me. But right now I can’t even read from a simple IOStream, even though everywhere I read it seems possible. I can’t even find a work-around where I load the file to an array, I really need to load from a file via filename, it seems…

5

I can reproduce your problem, using Julia 1.12.1 and JLD2 v0.6.2. Running the following script:

using JLD2

hello = "world"
save_object("example.jld2", hello)
hello_loaded = load_object("example.jld2")
load_object(open("example.jld2")) # this line fails

results in the error:

julia> include("test.jl")
ERROR: LoadError: FieldError: type IOStream has no field `readable`, available fields: `handle`, `ios`, `name`, `mark`, `lock`, `_dolock`
Stacktrace:
  [1] getproperty
    @ ./Base_compiler.jl:54 [inlined]
  [2] jldopen(io::IOStream, writable::Bool, create::Bool, truncate::Bool; plain::Bool, compress::Bool, typemap::Function)
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/JLD2.jl:312
  [3] jldopen(io::IOStream, writable::Bool, create::Bool, truncate::Bool)
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/JLD2.jl:304
  [4] jldopen(fname::IOStream, mode::String; iotype::Type, kwargs::@Kwargs{})
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/JLD2.jl:299
  [5] jldopen(fname::IOStream, mode::String)
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/JLD2.jl:290
  [6] jldopen(::Function, ::IOStream, ::Vararg{Any}; kws::@Kwargs{})
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/loadsave.jl:2
  [7] jldopen(::Function, ::IOStream, ::String)
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/loadsave.jl:1
  [8] load_object(filename::IOStream)
    @ JLD2 ~/.julia/packages/JLD2/WDhXU/src/loadsave.jl:248
  [9] top-level scope
    @ ~/tmp/test.jl:5
 [10] include(mapexpr::Function, mod::Module, _path::String)
    @ Base ./Base.jl:307
 [11] top-level scope
    @ REPL[1]:1
in expression starting at /home/ufechner/tmp/test.jl:5

I executed this test on Ubuntu 24.04.

Did you try the function load instead of load_object ?

UPDATE:
OK, load does not help.

Did you try jldopen ?

6

It’s suboptimal that this errors as it does. That in itself is probably worth an issue.

Check out JLD2.jl/test/wrapped_io.jl at master · JuliaIO/JLD2.jl · GitHub
for tests of what is implemented, and check out
JLD2.jl/src/io/io_wrappers.jl at master · JuliaIO/JLD2.jl · GitHub
for a discussion of requirements for IO types to be supported in JLD2.

Your current best bet would be to decompress your file into an IOBuffer (i.e. decompressing to Vector{UInt8} and wrapping that in an IOBuffer like so:

julia> using JLD2

julia> jldsave("test.jld2"; a=collect(1:10))

julia> bytes = read("test.jld2")
884-element Vector{UInt8}:
 0x48
 0x44
....
julia> bytes = read("test.jld2");

julia> io = IOBuffer(bytes)
IOBuffer(data=UInt8[...], readable=true, writable=false, seekable=true, append=false, size=884, maxsize=Inf, ptr=1, mark=-1)

julia> jldopen(io) do f
       f["a"]
       end
10-element Vector{Int64}:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10

Consider this as highly experimental features. PRs to improve API, usability, docs, or even performance are very welcome.
I had seen very little interest in these features the last few years so little effort went into developing these beyond a simple proof of concept implementation.

Note that a streaming implementation will never be useful. JLD2 is a hierarchical format that needs random access to the file to decode it. There is no way to efficiently make use of a streaming API and therefore I would prefer not to add one.

2 Likes