# Variational Inference of Mixture models?

**URL:** <https://discourse.julialang.org/t/variational-inference-of-mixture-models/40031>\
**Category:** Probabilistic Programming\
**Tags:** turing\
**Created:** [May 23, 2020, 4:33pm UTC](https://discourse.julialang.org/t/variational-inference-of-mixture-models/40031 "2020-05-23T16:33:34Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![marty0801](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marty0801/32/12715_2.png) [@marty0801](https://discourse.julialang.org/u/marty0801)\
**Post date:** [May 23, 2020, 4:33pm UTC](https://discourse.julialang.org/t/variational-inference-of-mixture-models/40031/1 "2020-05-23T16:33:34Z")

</div>

I’m just cutting my teeth on Turing. (BTW, thanks for this amazing software.) Does Turing support VI of mixture models? I attempted advi against a gaussian mixture model, but without success as below. The error occurred at ` Z[ii] ~ Categorical(wtrue)`. Is it possible? Thanks!

```julia
using Distributions
using Turing, MCMCChains

# make causal distribution
ncomptrue = 3
mutrue = [1,2,3] # ground-truth component means
muprior = Normal(0, 3)
sigma = 0.1 # a-priori known 
nsamples = 100
# components have equal probability
wtrue = [1,1,1]/ncomptrue
mixtrue = MixtureModel(Normal, [(mm, sigma) for mm in mutrue], wtrue)
Y = rand(mixtrue, nsamples)

# fit distribution with known # components
@model Mixmodel(x) = begin
    N = length(x)
    mu1 ~ muprior
    mu2 ~ muprior
    mu3 ~ muprior
    mu = [mu1, mu2, mu3]
    Z = Vector{Int}(undef, N)
    for ii in 1:N
        Z[ii] ~ Categorical(wtrue)
        x[ii] ~ Normal(mu[Z[ii]], sigma)
    end
end

model = Mixmodel(Y)

advi = ADVI(10, 1000)
q = vi(model, advi)

```

---

<div class="post-metadata">

**Author:** ![trappmartin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/trappmartin/32/1165_2.png) [@trappmartin](https://discourse.julialang.org/u/trappmartin)\
**Post date:** [May 23, 2020, 6:12pm UTC](https://discourse.julialang.org/t/variational-inference-of-mixture-models/40031/2 "2020-05-23T18:12:55Z")

</div>

Hi, you can use variational inference only for continuous parameters. In variational inference, the goal is to minimise the KL-divergence from the variational distribution to the true posterior. For this, we use gradient based optimisation in Turing, which requires the computation of gradients.

However, you can reformulate a mixture model such that it contains only continuous parameters.

The following model runs using ADVI.

```julia
@model function gmm(x, K)
    N = length(x)
    μ ~ filldist(Normal(0,1), K)
    w ~ Dirichlet(K, 1)
    for i in 1:N
        x[i] ~ MixtureModel(Normal, μ, w)
    end
end

```

Alternatively, you could also use the `arraydist` function, i.e.

```julia
@model function gmm(x, K)
    N = length(x)
    μ ~ filldist(Normal(0,1), K)
    w ~ Dirichlet(K, 1)
    x ~ arraydist(map(i -> MixtureModel(Normal, μ, w), 1:N))
end

```

Note that you might want to change the AD backend if you have a model with a lot of parameters. In this case it is probably not necessary, but for more involved models you might want to keep this in mind and possibly switch to a reverse mode AD. You can find details about this in the documentation of Turing.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [May 23, 2020, 6:50pm UTC](https://discourse.julialang.org/t/variational-inference-of-mixture-models/40031/3 "2020-05-23T18:50:31Z")

</div>

> [@trappmartin](#):
>
> `arraydist(map(i -> MixtureModel(Normal, μ, w), 1:N))`

`filldist(MixtureModel(Normal, μ, w), N)` is better in this case since the distribution is the same for all elements.

---

<div class="post-metadata">

**Author:** ![marty0801](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marty0801/32/12715_2.png) [@marty0801](https://discourse.julialang.org/u/marty0801)\
**Post date:** [May 23, 2020, 7:25pm UTC](https://discourse.julialang.org/t/variational-inference-of-mixture-models/40031/4 "2020-05-23T19:25:55Z")

</div>

Yes of course! Thank you very much!

---

<div class="post-metadata">

**Author:** ![marty0801](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marty0801/32/12715_2.png) [@marty0801](https://discourse.julialang.org/u/marty0801)\
**Post date:** [May 23, 2020, 9:22pm UTC](https://discourse.julialang.org/t/variational-inference-of-mixture-models/40031/5 "2020-05-23T21:22:48Z")

</div>

One follow on question. If you could please explain why K is required in the function signature. Consider the two nearly-identical models (the only difference being whether K is given as a keyword argument.)

```julia
@model gmm(x, K=3) = begin
    N = length(x)
    μ ~ filldist(Normal(0,1), K)
    w ~ Dirichlet(K, 1)
    for ii in 1:N
        x[ii] ~ MixtureModel(Normal, μ, w)
    end
end

@model gmm2(x) = begin
    K = 3
    N = length(x)
    μ = filldist(Normal(0,1), K)
    w ~ Dirichlet(K, 1)
    for ii in 1:N
        x[ii] ~ MixtureModel(Normal, μ, w)
    end
end

```

The following code executes successfully

```julia
model = gmm(Y)
advi = ADVI(10, 1000)
q = vi(model, advi)

```

Changing the first line above to `model=gmm2(Y)` gives an error. How can this be when `gmm(Y)` and `gmm2(Y)` appear to be functionally equivalent?

---

<div class="post-metadata">

**Author:** ![trappmartin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/trappmartin/32/1165_2.png) [@trappmartin](https://discourse.julialang.org/u/trappmartin)\
**Post date:** [May 24, 2020, 6:53am UTC](https://discourse.julialang.org/t/variational-inference-of-mixture-models/40031/6 "2020-05-24T06:53:36Z")

</div>

This should not make any difference. I cannot try it atm, but have you observed the same behaviour on a fresh Julia session?

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [May 24, 2020, 8:30am UTC](https://discourse.julialang.org/t/variational-inference-of-mixture-models/40031/7 "2020-05-24T08:30:53Z")

</div>

> [@marty0801](#):
>
> `μ = filldist(Normal(0,1), K)`

This should be `μ ~ filldist(Normal(0,1), K)`.

---

<div class="post-metadata">

**Author:** ![marty0801](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marty0801/32/12715_2.png) [@marty0801](https://discourse.julialang.org/u/marty0801)\
**Post date:** [May 24, 2020, 11:43am UTC](https://discourse.julialang.org/t/variational-inference-of-mixture-models/40031/8 "2020-05-24T11:43:58Z")

</div>

😳 Yep; that was the problem…

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [December 16, 2021, 10:06am UTC](https://discourse.julialang.org/t/variational-inference-of-mixture-models/40031/9 "2021-12-16T10:06:31Z")

</div>

For people stumbling into this post at a later moment. I wrote an extensive blog post about exactly this model; partly my post was based on this thread as well: [Bayesian Latent Profile Analysis (mixture modeling) - Huijzer.xyz](https://huijzer.xyz/posts/latent/)

In the post, I show that mixture models are very tricky to get right due to the label identification problem. To fix this, the post uses an ordered constraint like the ordered prior which is available in Stan.