Rescuing old Julia 0.5 Nullable{Date} type in 0.6

acunningham · June 20, 2018, 11:11am

Hi,

I’m upgrading to from Julia 0.5 to Julia 0.6 but I’m having some problems reading old JLD files with Nullable{Date} field.

Julia 0.5 type below:

type foo5
    id::Int64
    rollup_date::Nullable{Date}
end

Julia 0.6 type:

mutable struct foo5
    id::Int64
    rollup_date::Nullable{Date}
    
end

Following the JLD docs for rescuing old types I have the following:

type OldNullable{T}
    isnull::Bool
    value::T
end

JLD.readas(x::OldNullable) = Nullable(x.value, !x.isnull)
JLD.translate("Base.Nullable{Base.Dates.Date}",  "OldNullable{Base.Dates.Date}")

path = "c:\\foo5.jld"
file_dir, file_name = splitdir(path)
file_prefix, file_ext = splitext(file_name)
obj = JLD.load(path, file_prefix)

After trying to load this type I get the following error:

stored type foo5 does not match currently loaded type

Stacktrace:
 [1] jldatatype(::JLD.JldFile, ::HDF5.HDF5Datatype) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\jld_types.jl:721
 [2] read(::JLD.JldDataset) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:368
 [3] read(::JLD.JldFile, ::String) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:344
 [4] (::JLD.##41#42{String})(::JLD.JldFile) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:1241
 [5] #jldopen#11(::Array{Any,1}, ::Function, ::JLD.##41#42{String}, ::String, ::Vararg{String,N} where N) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:243
 [6] load(::FileIO.File{FileIO.DataFormat{:JLD}}, ::String) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:1240
 [7] #load#13(::Array{Any,1}, ::Function, ::String, ::String, ::Vararg{String,N} where N) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\FileIO\src\loadsave.jl:113
 [8] load(::String, ::String) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\FileIO\src\loadsave.jl:113

Any ideas or anything obvious that I’m doing wrong?

Regards,
Adrian.

rdeits · June 20, 2018, 2:59pm

Nullables are immutable, so perhaps your OldNullable should be a struct (the new equivalent of the old immutable) instead. I realize that your code matches the readme, but it’s possible that the readme is incorrect. See https://github.com/JuliaIO/JLD.jl/issues/180 where the version that worked used a struct for OldNullable.

acunningham · June 21, 2018, 10:33am

hmm…tried that, still doesn’t seem to work.

If I do exactly as the link states, I can write a 0.5 Nullable{Date} to disk and read in 0.6. But it doesn’t work when I have it nested inside my type.

pearlzli · June 25, 2018, 6:19pm

Could you try also defining an OldFoo5 type and corresponding readas method and translate call?

acunningham · June 25, 2018, 7:45pm

Yep, tried that, still does not work:

import JLD

struct OldNullable{T}
    isnull::Bool
    value::T
end

JLD.readas(x::OldNullable) = Nullable(x.value, !x.isnull)
JLD.translate("Base.Nullable{Base.Dates.Date}",  "OldNullable{Base.Dates.Date}")

mutable struct foo5 
    id::Int64
    rollup_date::Nullable{Date}
end

mutable struct oldfoo5
    id::Int64
    rollup_date::Nullable{Date}
end

JLD.readas(x::oldfoo5) = foo5(x.id, x.rollup_date)
JLD.translate("foo5", "oldfoo5")


path = "c:\\foo5.jld"
file_dir, file_name = splitdir(path)
file_prefix, file_ext = splitext(file_name)
obj = JLD.load(path, file_prefix)

gives the same error:

stored type oldfoo5 does not match currently loaded type

Stacktrace:
 [1] jldatatype(::JLD.JldFile, ::HDF5.HDF5Datatype) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\jld_types.jl:721
 [2] read(::JLD.JldDataset) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:368
 [3] read(::JLD.JldFile, ::String) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:344
 [4] (::JLD.##41#42{String})(::JLD.JldFile) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:1241
 [5] #jldopen#11(::Array{Any,1}, ::Function, ::JLD.##41#42{String}, ::String, ::Vararg{String,N} where N) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:243
 [6] load(::FileIO.File{FileIO.DataFormat{:JLD}}, ::String) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:1240
 [7] #load#13(::Array{Any,1}, ::Function, ::String, ::String, ::Vararg{String,N} where N) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\FileIO\src\loadsave.jl:113
 [8] load(::String, ::String) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\FileIO\src\loadsave.jl:113

pearlzli · June 25, 2018, 9:39pm

I think I got your example to work when I did the following:

Change the type of rollup_date in the oldfoo5 definition to OldNullable{Date}
Change the definition of readas on oldfoo5 to
readas(x::oldfoo5) = foo5(x.id, JLD.readas(x.rollup_date))

Does that work for you?

acunningham · June 26, 2018, 11:17am

Yes, that works just fine. Never thought to try that.

Many thanks!

acunningham · July 23, 2018, 9:30pm

So, I’ve encountered another issue with this. Reading in the old types works as suggested, however if I decide to create a new one and write it out, I can’t load it back in again. Have you encountered that?

So if you keep the existing code to “rescue” the old types but then create a new one and write it to disk. You’d can’t load it again. Same error then reappears.

Seems like the mapping to read OldNullable then causes in issue when trying to write out a regular Nullable.

However, i can load the type if I create a session and don’t include the mapping code.

new_foo = foo5(66, Date())

file_path = "C:\\new_foo.jld"

file_dir, file_name = splitdir(file_path)
file_prefix, file_ext = splitext(file_name)

JLD.save(file_path, file_prefix, new_foo, compress=false)

file_path = "C:\\new_foo.jld"
file_dir, file_name = splitdir(file_path)
file_prefix, file_ext = splitext(file_name)
obj = JLD.load(file_path, file_prefix)


stored type OldNullable{Base.Dates.Date} does not match currently loaded type

Stacktrace:
 [1] jldatatype(::JLD.JldFile, ::HDF5.HDF5Datatype) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\jld_types.jl:721
 [2] jldatatype(::JLD.JldFile, ::HDF5.HDF5Datatype) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\jld_types.jl:711
 [3] read(::JLD.JldDataset) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:368
 [4] read(::JLD.JldFile, ::String) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:344
 [5] (::JLD.##41#42{String})(::JLD.JldFile) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:1241
 [6] #jldopen#11(::Array{Any,1}, ::Function, ::JLD.##41#42{String}, ::String, ::Vararg{String,N} where N) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:243
 [7] load(::FileIO.File{FileIO.DataFormat{:JLD}}, ::String) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:1240
 [8] #load#13(::Array{Any,1}, ::Function, ::String, ::String, ::Vararg{String,N} where N) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\FileIO\src\loadsave.jl:113
 [9] load(::String, ::String) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\FileIO\src\loadsave.jl:113