# Gaussian Process Model with Turing

**URL:** <https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453>\
**Category:** Probabilistic Programming\
**Tags:** regression, turing, gaussian-process\
**Created:** [July 2, 2020, 9:10pm UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453 "2020-07-02T21:10:36Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [October 1, 2020, 7:00pm UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/42 "2020-10-01T19:00:04Z")

</div>

This is pretty much code-gold. I have some regression methods that someone else might like using. Is there a place to put handy snippits of Prob. prog. models?

---

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [October 1, 2020, 7:06pm UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/43 "2020-10-01T19:06:05Z")

</div>

> [@anon92994695](#):
>
> This is pretty much code-gold.

That’s very kind, thank you 🙂

> [@anon92994695](#):
>
> Is there a place to put handy snippits of Prob. prog. models?

For Soss models, I’d love to flesh out [GitHub - cscherrer/SossExamples.jl](https://github.com/cscherrer/SossExamples.jl) . PRs welcome!

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [October 1, 2020, 7:15pm UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/44 "2020-10-01T19:15:06Z")

</div>

> [@opera\_malenky](#):
>
> The latter part of the book covers (essentially) random effects GPs with this kind of structure

Thanks! That does look useful. I’ve been working my way through [this one](http://www.gaussianprocess.org/gpml/chapters/) (which has the benefit that it’s free), but the math quickly goes over my head and I’m finding it somewhat challenging to go from book examples into practical code (though I have to say, the design of these packages does make it easier to fiddle and get something working even if I only have a vague idea why)

---

<div class="post-metadata">

**Author:** ![trappmartin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/trappmartin/32/1165_2.png) [@trappmartin](https://discourse.julialang.org/u/trappmartin)\
**Post date:** [October 2, 2020, 10:29am UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/45 "2020-10-02T10:29:34Z")

</div>

Carl’s book is a really good introduction to GPs in my opinion.

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [October 2, 2020, 2:24pm UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/46 "2020-10-02T14:24:52Z")

</div>

Yeah, it’s far more comprehensible than many other things I’ve looked at. The deficit is entirely on me, and the fact that the last math class I took was during my freshman year of college… 18 years ago… 😱

---

<div class="post-metadata">

**Author:** ![nucklass](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nucklass/32/11989_2.png) [@nucklass](https://discourse.julialang.org/u/nucklass)\
**Post date:** [October 5, 2020, 11:50pm UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/47 "2020-10-05T23:50:36Z")

</div>

Question for the KernelFunctions/AbstractGP authors, though if anyone else has any idea feel free to chime in too. Whats the usual procedure for diagnosing/fixing PosDef Errors during estimation of hyperparameters? I’ve got a fairly large model (~6000 observations) and it’s seems somewhat random if I get the error or not, so I’ve just been repeating dynamicHMC sample calls until it doesn’t error. Obviously this is not sustainable and rather uniformed. Is the conventional wisdom just to increase the value added to the diagonal of the covariance matrix until I stop getting errors or should is this an indication that my choice of prior for hyperparameters is off? Again, awesome work on both packages!

---

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [October 5, 2020, 11:56pm UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/48 "2020-10-05T23:56:27Z")

</div>

It’s not clear to me whether a singular covariance is a problem for the model, or just for the way it’s currently being computed. Maybe [PositiveFactorizations.jl](https://github.com/timholy/PositiveFactorizations.jl) could help with this?

---

<div class="post-metadata">

**Author:** ![nucklass](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nucklass/32/11989_2.png) [@nucklass](https://discourse.julialang.org/u/nucklass)\
**Post date:** [October 6, 2020, 1:37am UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/49 "2020-10-06T01:37:31Z")

</div>

Ah actually looks like the KernelFunctions team already answered my question. Check out [https://github.com/JuliaGaussianProcesses/KernelFunctions.jl/blob/6bca6a17284dcd12fc4eca945435697e50a524b3/src/matrix/kernelpdmat.jl](https://github.com/JuliaGaussianProcesses/KernelFunctions.jl/blob/6bca6a17284dcd12fc4eca945435697e50a524b3/src/matrix/kernelpdmat.jl)  
if anyone else is running into this issue. Still errors after switching from kernelmatrix to kernelpdmat in the model specification, but at least the error message is much more informative 😅 . Looks like I might need to fiddle with some priors or reformat the data to make the feature matrix more sparse.

---

<div class="post-metadata">

**Author:** ![theogf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/theogf/32/1987_2.png) [@theogf](https://discourse.julialang.org/u/theogf)\
**Post date:** [October 8, 2020, 8:33pm UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/50 "2020-10-08T20:33:46Z")

</div>

Hey @nucklass, one of the authors here. I designed kernelpdmat exactly for this purpose but it can still not be enough for some situations. What you might want to look for is if your kernel matrix is low-rank, i.e. you have some columns which are numerically identical. This is usually due to having 2 observations extremely close to each other, This closeness can happen for example with very large lengthscales for instance (I imagine this is what is happening in your optimization). To be honest there is no really good solution for this… Sparse GPs tend to avoid this problem and given that you have a lot of observations this might be the way to go.  
EDIT: We should probably write a full paragraph about the “good GP practices” somewhere in some docs 😃

---

<div class="post-metadata">

**Author:** ![nucklass](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nucklass/32/11989_2.png) [@nucklass](https://discourse.julialang.org/u/nucklass)\
**Post date:** [October 8, 2020, 10:26pm UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/51 "2020-10-08T22:26:28Z")

</div>

Thanks for the reply!

> [@theogf](#):
>
> Sparse GPs tend to avoid this problem and given that you have a lot of observations this might be the way to go.

This is looking more and more like the direction I should go. I’m only familiar with the simplest sparse approximations, ie Subsets of Data or Subsets of Regressors. Would it be entirely foolish to try and combine sparse approximations and the use of MCMC sampling of kernel hyperparamters? I.E. try and extend the Soss + Turing programs above with some sparse approximation of the latent function. From a cursory glance it seems like quite a few of Sparse methods attempt to optimize hyperparameters and inducing points simultaneously, and I’m not quite sure what estimation methods are being used in each instance. From a performance standpoint, I run a lot of Bayesian models so waiting a few days for sampling to finish is nothing new. Running an analysis that some one could reproduce/rerun is rather important, though. Also if Augmented Gaussian Processes implements any of this already, thanks in advance.

---

<div class="post-metadata">

**Author:** ![ElOceanografo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eloceanografo/32/624_2.png) [@ElOceanografo](https://discourse.julialang.org/u/ElOceanografo)\
**Post date:** [October 8, 2020, 11:30pm UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/52 "2020-10-08T23:30:31Z")

</div>

FYI, theres a [PR](https://github.com/willtebbutt/Stheno.jl/pull/136) to Stheno right now that will let you use Sparse GPs in Turing. Not battle-tested yet, but I’ve got it to run for some simple problems. And here’s the thread that led to it, if you didn’t see it:

> [@Implementing sparse Gaussian process model with Turing and Stheno](https://discourse.julialang.org/t/implementing-sparse-gaussian-process-model-with-turing-and-stheno/46886/7):
>
> Okay, I came back to this and after some hacking have got the following to run and produce basically sensible results. struct SparseFiniteGP{T1\<:Stheno.FiniteGP, T2\<:Stheno.FiniteGP} \<: Distributions.AbstractMvNormal fobs::T1 finducing::T2 end function Stheno.logpdf(f::T, y::AbstractArray{T1, 1}) where {T \<: SparseFiniteGP, T1 \<: Real} return elbo(f.fobs, y, f.finducing) end Base.length(f::SparseFiniteGP) = length(f.fobs) Distributions.\_rand!(rng::Random.\_GLOBAL\_RNG, f::SparseFini…

---

<div class="post-metadata">

**Author:** ![nucklass](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nucklass/32/11989_2.png) [@nucklass](https://discourse.julialang.org/u/nucklass)\
**Post date:** [October 9, 2020, 12:55am UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/53 "2020-10-09T00:55:13Z")

</div>

Oh that’s very cool! I’ll definitely give this a try. I’ve heard Stheno’s kernel interface is going to switch over to KernelFunctions.jl fairly soon, so I guess everything’s coming together 😄.

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [November 5, 2020, 4:08pm UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/54 "2020-11-05T16:08:08Z")

</div>

Alright, I’m back! This is my focus until I get something working. To reiterate the goals here, because this post has meandered a bit (which is great, I’m learning a ton!), the situation is this:

- I have longitudinal patient biosamples. That is, I’ve taken measurements from multiple people (`subject`s) over time.
- Many of the measurements are continuous, in this case in particular, I’m looking at the dependent variable `bug`, and one of the predictor variables is `nutrient`
- `bug` typically changes over time, but not in a single direction. We expect measurements for a given subject to be highly correlated for samples close in time, less so for samples more distant in time.
- we expect measurements from different `subject`s to be completely independent of one another (that is, there’s no relationship between `bug1` for `subj1` at `date1` and `bug1` for `subj2` at `date1`. Akin to a random effect.
- There are other categorical and boolean variables that do not tend to change for a particular subject (eg gender, obesity)

We would like to model the measurement of `bug` over time as a gaussian process, where (in the language of LMs) things like `obese` are fixed effects, `subject` is a random effect, and the covariance of `nutrient` over `time` is squared exponential.

After doing some additional reading, I think what I really want is to encode at least `subject`, and possibly `obese` as index variables, a la [https://github.com/TuringLang/Turing.jl/pull/1422](https://github.com/TuringLang/Turing.jl/pull/1422). But I’m having issues figuring this out, and I think it has to do with a misunderstanding of how different things interact. So first things first, I tried building a slightly extended version of the `demo` model in #1422, such that the `g` variable is an index variable, and X is a matrix of typical continuous variables (also using the [linreg tutorial](https://turing.ml/dev/tutorials/5-linearregression/) as inspiration. This seems to work:

```julia
# Dummy data
x = vcat(randn(10), randn(10).+10, randn(10).-5)
y = vcat(randn(10), randn(10).+10, randn(10).-5) .* 2
g = vcat(ones(Int, 10), ones(Int, 10)*2, ones(Int, 10)*3)

@model function demo(x, y, g)
    ng = length(unique(g))
    # group coefficients priors
    a ~ filldist(Normal(), ng)
    
    # x coefficient prior
    b ~ Normal()

    mu = a[g] + b * x
    
    y .~ Normal.(mu)
end

model = demo(x,y,g);
sample(model, NUTS(0.8), 1_000)

```

```julia
Summary Statistics
  parameters mean std naive_se mcse ess rhat  
      Symbol Float64 Float64 Float64 Float64 Float64 Float64  
                                                                            
        a[1] -0.6291 0.2943 0.0132 0.0117 380.2325 1.0030  
        a[2] 2.6719 0.8799 0.0394 0.0438 174.4973 1.0048  
        a[3] -2.0860 0.4941 0.0221 0.0229 175.6049 1.0010  
           b 1.7736 0.0921 0.0041 0.0048 167.5996 1.0039

```

So now imagine I have a `tp` argument that includes timepoints. How do I incorporate the above specification into a GP? The following code runs, but I’m not 100% sure that it’s actually doing what I’m intending…

```julia
tp = repeat(1:10, 3)

sekernel(alpha, rho) = 
  alpha^2 * KernelFunctions.transform(SEKernel(), sqrt(0.5)/rho)

@model function demo(x, y, g, tp, jitter=1e-6)
    ng = length(unique(g))
    # group coefficients priors
    a ~ filldist(Normal(), ng)
    
    # x coefficient prior
    b ~ Normal()

    # gp priors
    sig2 ~ LogNormal(0, 1)
    alpha ~ LogNormal(0, 0.1)
    rho ~ LogNormal(0, 1)

    # lm specs
    mu = a[g] + b * x
    
    # gp specs
    kernel = sekernel(alpha, rho) # covariance function
    K = kernelmatrix(kernel, reshape(tp, length(tp), 1), obsdim=1) # cov matrix
    K += LinearAlgebra.I * (sig2 + jitter)
    
    y .~ MvNormal(mu, K)
end

model = demo(x,y,g,tp);
sample(model, NUTS(0.8), 1_000)

```

```julia
Summary Statistics
  parameters mean std naive_se mcse ess rhat  
      Symbol Float64 Float64 Float64 Float64 Float64 Float64  
                                                                            
        a[1] -0.4016 0.7114 0.0318 0.0164 746.6413 0.9994  
        a[2] 0.5394 0.9962 0.0446 0.0565 430.2212 0.9982  
        a[3] -0.6955 0.8198 0.0367 0.0488 503.1865 0.9993  
       alpha 0.9929 0.0944 0.0042 0.0012 653.7770 0.9980  
           b 1.9989 0.1195 0.0053 0.0074 382.8640 0.9982  
         rho 1.7239 2.2199 0.0993 0.0645 376.1499 1.0025  
        sig2 5.9109 1.6832 0.0753 0.0698 497.3395 0.9982  

```

Does this model make sense / fit what I described above?

This also runs on my real data (no `PosDefException`, yay!) where the number of unique `g` is ~300), but is suuuuper slow. Any suggestions on this front would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [November 17, 2020, 12:04am UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/55 "2020-11-17T00:04:53Z")

</div>

Bump. Also some updates.

I’ve been trying to do a bit of benchmarking and optimizing on the model above, following the [performance tips](https://turing.ml/dev/docs/using-turing/performancetips). I’ve already avoided loops, and when I do `@code_warntype model = demo(x,y,g,tp)`, everything is blue (though this is my first time using that macro, not sure that’s right).

Then I experimented with different backends. First thing to note - `Zygote` gives me an error (something about not being able to iterate `nothing`). I’m assuming this has to do [with this statement](https://turing.ml/dev/docs/using-turing/performancetips#special-care-for-tracker-and-zygote):

> In case of `:tracker` and `:zygote` , it is necessary to avoid loops for now. This is mainly due to the reverse-mode AD backends `Tracker` and `Zygote` which are inefficient for such cases. `ReverseDiff` does better but vectorized operations will still perform better.

I don’t explicitly use a loop, is it possible that the GP is using one implicitly?

My real data is 921 rows, each subject has 1-4 timepoints, and there are 305 total subjects. When I do the first 50 rows, 100 samples for the chain and `forwarddiff`:

- `NUTS(0.8)` takes 17.6 sec
- `HMC(0.1, 10)` takes 12.9 sec
- `HMC(0.1, 20)` takes 6.7 sec

The first 50 rows has 19 subjects. When I bump up to 100 rows (36 subjects), that last one takes 51 sec (almost 10 times as long), and using `reversediff` takes about 10.5 min. At 150 rows (53 subjects), it’s 230 sec. If things keep scaling like this, I’m looking at days for the whole set. Is there a better way to do this?

I also made an attempt to do what was suggested earlier and add a separate subject GP with a really short length scale - it’s possible I did that wrong, but it did not improve things much.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 17, 2020, 6:31am UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/56 "2020-11-17T06:31:41Z")

</div>

> [@kevbonham](#):
>
> When I do the first 50 rows, 100 samples for the chain and `forwarddiff` :

Note that the relevant benchmark is probably not time/sample, but time/effective sample size.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [November 17, 2020, 6:37am UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/57 "2020-11-17T06:37:32Z")

</div>

Sorry to hijack this thread, but this is moving towards a more general question I’ve had for Bayesian modelling in general and Turing specifically for a while: how can I form a reasonable expectation about how long a model will take to estimate in advance? Is it about the number of observations, the number of parameters, or the structure of the model itself? And what is effective sample size in this context?

For someone only trained in classical econometrics, where one estimates mostly linear models that take a fraction of a second to estimate, it’s hard to guess when I could feasibly switch my analysis strategy over to Bayesian.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 17, 2020, 6:42am UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/58 "2020-11-17T06:42:42Z")

</div>

> [@nilshg](#):
>
> how can I form a reasonable expectation about how long a model will take to estimate in advance?

Generally,

1. evaluation time of the posterior (with gradients, if you are using those), which does depend on the number of observations etc (unless you have summary statistics),
2. how well-behaved the log posterior is (eg non-uniform curvature is difficult to deal with)

See [the Stan manual](https://mc-stan.org/docs/2_21/reference-manual/effective-sample-size-section.html) for an explanation of ESS.

---

<div class="post-metadata">

**Author:** ![theogf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/theogf/32/1987_2.png) [@theogf](https://discourse.julialang.org/u/theogf)\
**Post date:** [November 17, 2020, 3:12pm UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/59 "2020-11-17T15:12:12Z")

</div>

For example in this situation, a kernel matrix `K` of size `length(tp) x length(tp)` is created and passed to `MvNormal`. Now `MvNormal` needs to invert this matrix with complexity `length(tp)^3`, which is usually the bottleneck with GPs.  
So basically every sampling step has at least complexity `N^3`, which, for `N=300` is not negligible!

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [November 17, 2020, 7:15pm UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/60 "2020-11-17T19:15:21Z")

</div>

> [@Tamas\_Papp](#):
>
> Note that the relevant benchmark is probably not time/sample, but time/effective sample size.

Note that I don’t really understand the distinction 😂

> [@theogf](#):
>
> So basically every sampling step has at least complexity `N^3` , which, for `N=300` is not negligible!

Yeah, this doesn’t seem great 😩. In this case, I really only care about the GP and linear hyperparameters, not the subject-specific stuff (`a[g]` in the model). I found a presentation from a former colleague, where he wrote that we want to

> - approximate the latent posterior and integrate it out.
> - approximation by laplace method
> - use saddle-free Newton to avoid saddle nodes

I did some googling and didn’t find any julia-specific information about this - any ideas where I should look?

---

<div class="post-metadata">

**Author:** ![mschauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mschauer/32/13946_2.png) [@mschauer](https://discourse.julialang.org/u/mschauer)\
**Post date:** [November 18, 2020, 9:26am UTC](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453/61 "2020-11-18T09:26:54Z")

</div>

@kevbonham Can you restate the actual model you are interested in?

[Previous page](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453.md?page=2)

[Next page](https://discourse.julialang.org/t/gaussian-process-model-with-turing/42453.md?page=4)
