Integrating a long(ish) running simulation into Pluto

Tooling Pluto
1

Hi,

quick question: I have a simulation model that takes in the order of 30-60 seconds to run. (This one if you’re interested).

If I wanted to embed it in a Pluto notebook, what’s the best practice? Can I detach it somehow from Pluto’s normal reactive behaviour? Is there something I can use as a progress indicator?

Graham

2

I very often write big simulations in Pluto.

The basic tip is that generally your long simulation should be called from a function and not a script (or a begin block or consecutive cells). Now you can have this long function defined in Pluto, but I usually do it differently.

What I commonly do is start Pluto in a local directory with local package manager using
Pkg.activate. I define a module in this directory and I put my non-Pluto bulky code (e.g. your simulation) in there. Because of Revise, I can also update it easily.

Then when I start Pluto the first cells do using Revise and Pkg.activate(".").
I import my local module using MyModule and I access the functions from there.

Can I detach it somehow from Pluto’s normal reactive behaviour?

No. If you change any variables that are used as arguments to call the simulation function in your notebook, then the simulation will rerun. But that’s desirable. If you structure your notebook nicely I don’t see why would that be a problem.

Regarding the progress bar, you can definitely use some inside you simulation function.
See ProgressMeter in Pluto
Pluto will then visualize it.

btw your link doesn’t work.

5 Likes
3

Using the temporary execution barrier feature in Pluto may help with the reactivity piece: Temporary Execution barrier by lungben · Pull Request #985 · fonsp/Pluto.jl · GitHub

2 Likes
4

is this the “Disable cell” button ?

3 Likes
5

See also Base.@async and Threads.@spawn. Those create tasks that are detached from the interative thread. You can then call fetch on the task, which will wait until a result is available, and give you that.

2 Likes
6

Thanks. That’s interesting and helpful.

7

FWIW, here is a macro I use if I deliberately want to save some output, in case I need to restart the notebook but don’t want to the simulation to recompute again. Note that this deliberately circumvents Pluto.jl consistency by using external state. So if you change something you need to consider that the values are not recomputed!
The snippet requires JLD2.jl for saving/loading and creates a folder ~/home/results/notebooks/<Notebookname>/ and saves the content of variables there. To save something you do:

@load_or_generate myvar = some_long_ass_computation(p1,p2,p3,p4)

Then each time the cell is run, the macro checks whether there is a save file myvar.jld2 in the corresponding folder and if yes just loads the contents. If you change the computation, you need to delete the file for the computation to refresh!

Code 
begin
	NOTEBOOKNAME = basename(replace(@__FILE__, r"\.jl#==#.*" => ""))
	function load_or_generate(f, name)
		path = joinpath(homedir(), "results/notebooks", NOTEBOOKNAME)
		name = endswith(name, ".jld2") ? name : name*".jld2"
		isdir(path) || mkpath(path)
		fullpath = joinpath(path, name)
		isfile(fullpath) && return jldopen(fullpath, "r")["data"]
		data = f()
		jldsave(fullpath; data)
		return data
	end
	macro load_or_generate(expr)
		expr.head != :(=) && error("@load_or_generate is only applicable to simple assignments of the form `a = b`")
		var, rhs = expr.args
		#eqsign == := 
		return quote
			$(esc(var)) = $(esc(:load_or_generate))($(string(var))) do
				return $(esc(rhs))
			end
		end
	end
	md"""Setup save to $(joinpath(homedir(), "results/notebooks", NOTEBOOKNAME))"""
end
2 Likes
8

While we are at it, and you really need to change something that pointlessly reruns the simulation, you can use memoization tools, to avoid that, e.g., Memoization.jl. But be really careful with the consistency of the results. That’s similar to @abraemer solution, but doesn’t work between new sessions.

2 Likes