Deserialization of Turing model fails in Quarto document

Specific Domains Probabilistic Programming
1

I have several results from an optimization serialized to disk. They are all NamedTuples result = (mdl = A Turing Model, chn = A Chains object) Now I want to return predictions from a model in my test.qmd file:

---
title: "Turing Serialization Test"
format: pdf
engine: julia
julia:
  exeflags: ["--project=@."]
  env: ["JULIA_NUM_THREADS=5"]
---

```{julia}
#| label: packages-scripts
using CSV, DataFrames, Turing, StatsBase, Random, Plots, StatsPlots
using ApproxFun, CategoricalArrays, NamedArrays, LaTeXStrings, Loess
using ADTypes, KernelDensity, Serialization, DynamicPPL, LinearAlgebra
using BenchmarkTools, IterTools, StatProfilerHTML, ReverseDiff
using Suppressor, Distributions
## using Enzyme

include("../src/utils.jl")
include("../src/estimation.jl")
include("../src/loadgermdata.jl")
include("../src/diag.jl")
include("../src/diagplots.jl")
include("../src/diagchain.jl")
include("../src/diagrbf.jl")
include("../src/norm.jl")
include("../src/rbf.jl")
include("../src/analysis.jl")

r = deserialize("output/optim30-50")[1]
returned(r.mdl.mdl, r.chn)

However, when doing quarto render test.qmd I get following error

UndefVarError: `Normal` not defined in `Main` Suggestion: check for spelling errors or missing imports. ```

Now, when putting the exact same code from the chunk into test.jl, it works. It also works when executing the code in test.qmd interactively and just sending it to the REPL. There are some Normal priors in the model. Any suggestions? I have no idea where to look for the error. For a simple toy model returned works just fine after deserialize.

2

Because Quarto runs stuff in a Jupyter notebook, maybe try putting the code in a notebook and see if it runs.

3

Because the YAML headers set

engine: julia

this file will not be routed through Jupyter for evaluation of the code blocks.

4

This sounds like a cache error. Are you sure you have turned cache off? Because if you cache then the packages do not get loaded in the next cell and your reference to Normal for example will be unknown. Can you post a more minimal example without all these includes?

5

I tried disabling caching in the YAML header but the problem persists. Regarding a more minimal example: Some of those includes could probably be removed, but I am not sure what this would show. I tried to reproduce the error with a simple toy model, but this worked.

I will try fitting and serializing the models anew and see if it works. If not, I will simply save the predictions to disk, next to model and chain.

6

So I got a little further. While playing around a bit with the deserialized object, I encountered The applicable method may be too new: running in world age 27386, while current world is 27388.

This prevented me from calling Turing.returned(r.mdl.mdl, r.chn) inside a function. On top level it worked fine. To call it inside a function I need to use invokelatest like here

function savepredictions(files)
    for f in files
        results = deserialize(f);
        res = NamedTuple[]
        for r in results
            p = invokelatest(Turing.returned, r.mdl.mdl, r.chn)[1]
            push!(res, (mdl = r.mdl, chn = r.chn, prd = p))
        end
        serialize(f, res)
    end
end

savepredictions(readdir("output"; join = true))

I think this is behind the error when compiling via quarto. Probably chunks in a quarto file are not really top level? No idea really.
invokelatest(Turing.returned, r.mdl.mdl, r.chn)[1] inside test.qmd still doesn’t work.