The results of my simulations are saved as DataFrames in JLD2 files:
# Running simulations and generating results...
sim = Sim(d, adf, model)
# convert struct to dict
sim2d = struct2dict(sim)
# save the dict to name, using DrWatson safesave
Until tonight, everything was working fine, but now I get the following warning when I try to load the data:
julia> log_file = jldopen(fnamef)
julia> adf = log_file["adf"]
┌ Warning: saved type DataFrame is missing field metadata in workspace type; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/HnW0g/src/data/reconstructing_datatypes.jl:164
DataFrames package is already loaded via the using command and DataFramedatatype is defined in the workspace. I am confused about the meaning of the warning, what could be the source of the problem and how to deal with it. I appreciate your comments and help.
I am working with Julia@1.8.2, JLD2@0.4.25 and DataFrames@1.4.1.
DataFrame is now a mutable struct and has three new fields
metadata, colmetadata, and allnotemetadata;
this change makes DataFrame objects serialized under earlier
versions of DataFrames.jl incompatible with
version 1.4 (#3055)
For anyone needing to convert DataFrame objects from 1.3 to 1.4 version (e.g. if you serialized your objects for short-term storage).
The easiest solution is to use Tables.columntable on DataFrame objects created under DataFrames.jl 1.3 version. Then serialize them. Next upgrade DataFrames.jl to 1.4. Deserialize the NamedTuple, and next transform it back to DataFrame.
Perhaps convert the dataframes to Tables using the old dataframe package and serialize them as Tables?
Then, using the new DataFrames package read the Tables from the jld2 file and convert them into the new DataFrames.
(note that the constructor for the DataFrame I used here explicitly states that it shouldn’t be used directly…)
What’s happening here is that the typemap argument tells JLD2 to watch out for stored structs
with the name "DataFrames.DataFrame" and should use a special reconstruction method, i.e.
load the fields as a named tuple and call convert to get a DataFrame.