Saving workspace to file

General Usage
1

Commonly I run a large set of simulations over night/weekend. All of this information exists in a Jupyter notebook and if the browser/jupyter crashes everything is lost and needs to be redone for the next day. After this happened this morning (again) I’ve decided I should be more robust n my approach.

The best would be if I could save my workspace, like I used to do in matlab. Then in case things go wrong I have a copy on the disk which I can use load up and use.

I have tried to understand how to do it, but found it surprisingly hard. There is a package called JLD, however I have read the documentation and it only describes saving variables, and not the entire workspace. Also it seems hard to use with non standard types, like the solutions to ODE/SDE problems, or the types I save my models in.

Is this possible to do, and if so, how?
cheers

1 Like
2

You could use JLD2:

https://github.com/simonster/JLD2.jl

@save <filename>

saves all variables in the global namespace to the file; @load loads them again.

6 Likes
3

Thank you, that is very helpful.

4

I have been trying through the day to get this to work, however I have had problems with non-trivial types. Typically what I want to save are solutions to Ordinary Differential Equations or Stochastic Differential Equations. I run this code

using JLD2
using DifferentialEquations
f(u,p,t) = 1.01*u
u0=1/2
tspan = (0.0,1.0)
prob = ODEProblem(f,u0,tspan)
sol = solve(prob,Tsit5(),reltol=1e-8,abstol=1e-8);
@lsave "tmpfile.jld"

and then turns the kernel off (I do everything in Jupyter by the way). So far everything is fine. Now I run:

using JLD2
using DifferentialEquations
using Plots
@load "tmpfile.jld"

This time I get this warning:

WARNING: type #f does not exist in workspace; reconstructing
WARNING: some parameters could not be resolved for type DiffEqBase.ODEProblem{Float64,Float64,false,Void,#f,Void,Void,UniformScaling{Int64},DiffEqBase.StandardODEProblem}; reconstructing
WARNING: some parameters could not be resolved for type OrdinaryDiffEq.InterpolationData{#f,Array{Float64,1},Array{Float64,1},Array{Array{Float64,1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}; reconstructing

which seems ominous. Finally I try

plot(sol)

and gets this plain error

type ##DiffEqBase.ODEProblem{Float64,Float64,false,Void,#f,Void,Void,UniformScaling{Int64},DiffEqBase.StandardODEProblem}#666 has no field p

Stacktrace:
 [1] macro expansion at /home/Torkel.Loman/.julia/v0.6/DiffEqBase/src/solutions/solution_interface.jl:99 [inlined]
 [2] apply_recipe(::Dict{Symbol,Any}, ::DiffEqBase.ODESolution{Float64,1,Array{Float64,1},Void,Void,Array{Float64,1},Array{Array{Float64,1},1},JLD2.ReconstructedTypes.##DiffEqBase.ODEProblem{Float64,Float64,false,Void,#f,Void,Void,UniformScaling{Int64},DiffEqBase.StandardODEProblem}#666,OrdinaryDiffEq.Tsit5,JLD2.ReconstructedTypes.##OrdinaryDiffEq.InterpolationData{#f,Array{Float64,1},Array{Float64,1},Array{Array{Float64,1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}#667}) at /home/Torkel.Loman/.julia/v0.6/RecipesBase/src/RecipesBase.jl:291
 [3] _process_userrecipes(::Plots.Plot{Plots.GRBackend}, ::Dict{Symbol,Any}, ::Tuple{DiffEqBase.ODESolution{Float64,1,Array{Float64,1},Void,Void,Array{Float64,1},Array{Array{Float64,1},1},JLD2.ReconstructedTypes.##DiffEqBase.ODEProblem{Float64,Float64,false,Void,#f,Void,Void,UniformScaling{Int64},DiffEqBase.StandardODEProblem}#666,OrdinaryDiffEq.Tsit5,JLD2.ReconstructedTypes.##OrdinaryDiffEq.InterpolationData{#f,Array{Float64,1},Array{Float64,1},Array{Array{Float64,1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}#667}}) at /home/Torkel.Loman/.julia/v0.6/Plots/src/pipeline.jl:81
 [4] _plot!(::Plots.Plot{Plots.GRBackend}, ::Dict{Symbol,Any}, ::Tuple{DiffEqBase.ODESolution{Float64,1,Array{Float64,1},Void,Void,Array{Float64,1},Array{Array{Float64,1},1},JLD2.ReconstructedTypes.##DiffEqBase.ODEProblem{Float64,Float64,false,Void,#f,Void,Void,UniformScaling{Int64},DiffEqBase.StandardODEProblem}#666,OrdinaryDiffEq.Tsit5,JLD2.ReconstructedTypes.##OrdinaryDiffEq.InterpolationData{#f,Array{Float64,1},Array{Float64,1},Array{Array{Float64,1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}#667}}) at /home/Torkel.Loman/.julia/v0.6/Plots/src/plot.jl:179
 [5] plot(::DiffEqBase.ODESolution{Float64,1,Array{Float64,1},Void,Void,Array{Float64,1},Array{Array{Float64,1},1},JLD2.ReconstructedTypes.##DiffEqBase.ODEProblem{Float64,Float64,false,Void,#f,Void,Void,UniformScaling{Int64},DiffEqBase.StandardODEProblem}#666,OrdinaryDiffEq.Tsit5,JLD2.ReconstructedTypes.##OrdinaryDiffEq.InterpolationData{#f,Array{Float64,1},Array{Float64,1},Array{Array{Float64,1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float64,Float64}}#667}) at /home/Torkel.Loman/.julia/v0.6/Plots/src/plot.jl:52

Further more, I repeat my test but using another special type, the reaction networks produced by the DiffEqBiological package. I use this code

using JLD2
using DifferentialEquations
rn_model = @reaction_network rnType begin
    1.0, X --> 0
end
@save "tmpfile.jld"

and get this response:

WARNING: skipping rn_model because it contains a pointer

How limited is JLD2 in what parts of the works pace I can actually successfully save? I succeed with the simpler types, but there should be some way to store also this kinds of stuff. Or is there nothing and I am running into a dead end?

1 Like