WAIC, or LOO etc in Turing?

Specific Domains Probabilistic programming
7

thanks Seth for a swift reply! I’m very happy to know how to get generated quantities out of a Turing model!

I wonder if obtaining the log-likelihood is only half the battle. I would much prefer to not have to create my own LOO-PSIS function, and rely instead upon ArviZ.

I’ve been trying this using the Stan interface. But to my surprise, even when I change the name of the generated quantity to log_likelihood (which ArviZ seems to expect), loo and waic seem not to detect it

using CmdStan
using ArviZ

set_cmdstan_home!(homedir() * "/cmdstan/")

J = 8
y = [28.0, 8.0, -3.0, 7.0, -1.0, 1.0, 18.0, 12.0]
sigma = [15.0, 10.0, 16.0, 11.0, 9.0, 11.0, 10.0, 18.0]
schools = [
    "Choate",
    "Deerfield",
    "Phillips Andover",
    "Phillips Exeter",
    "Hotchkiss",
    "Lawrenceville",
    "St. Paul's",
    "Mt. Hermon"
];

schools_code = """
data {
  int<lower=0> J;
  real y[J];
  real<lower=0> sigma[J];
}

parameters {
  real mu;
  real<lower=0> tau;
  real theta[J];
}

model {
  mu ~ normal(0, 5);
  tau ~ cauchy(0, 5);
  theta ~ normal(mu, tau);
  y ~ normal(theta, sigma);
}

generated quantities {
    vector[J] log_likelihood;
    vector[J] y_hat;
    for (j in 1:J) {
        log_likelihood[j] = normal_lpdf(y[j] | theta[j], sigma[j]);
        y_hat[j] = normal_rng(theta[j], sigma[j]);
    }
}
"""

schools_dat = Dict("J" => J, "y" => y, "sigma" => sigma)
stan_model = Stanmodel(
    model = schools_code,
    nchains = 4,
    num_warmup = 1000,
    num_samples = 1000,
)
_, stan_chns, _ = stan(stan_model, schools_dat, summary = false);

stan_infdata = convert_to_inference_data(stan_chns)

waic(stan_infdata)

This last line gives the same error as previously. Is there something obvious that I’m not getting?