I ran into a similar problem with JLD, it took over 20 minutes to load my data. I just wanted to leave this here if anybody runs into something similar, with JLD2 it only took 50 seconds to load the same data.
using JLD2
@time load("outfiles/aggregated_results.jld2")
49.773390 seconds (9.72 M allocations: 2.651 GiB, 7.02% gc time)
Dict{String,Any} with 7 entries: