ModelingToolkit & Distributions for MLE estimation

dmetivie
Tue, 18 Apr 2023 12:33:06 +0000
Optimization (Mathematical)

I am trying to use ModelingToolKit.jl to write a maximum loglikelihood estimator for generic distributions eg Univariate, Mixture, Multivariate etc.

using ModelingToolkit, Optimization, OptimizationOptimJL
using Distributions

f(θ) = MixtureModel([Exponential(θ[1]), Exponential(θ[2])], [θ[3], 1-θ[3]])
ℓ(θ, x) = -loglikelihood(f(θ), x) # = sum(logpdf(f(θ), y) for y in x)

# Data
N = 100
θtrue = [10, 1, 0.2]
x =  rand(f(θtrue), N)

# Observation as parameters (not sure if this is needed?)
@parameters y[1:N]

# variable
@variables θ[1:3]

mle = ℓ(θ, y)

# # Not tested yet, the code above need to work first
# @named sys = OptimizationSystem(mle, θ, y)
# θ0 = [θ[1] => 20, θ[2] => 4, θ[3] => 0.5]
# p = [y[i] => x[i] for i in eachindex(x)]

# prob = OptimizationProblem(sys, θ0, grad = true, hess = true)

However, this does not work:

ERROR: TypeError: non-boolean (Num) used in boolean context
....
[4] Exponential(θ::Num)
@ Distributions C:\Users\dmetivie\.julia\packages\Distributions\Spcmv\src\univariate\continuous\exponential.jl:29

I tried @register_symbolic Exponential(θ) but now the error is

ERROR: MethodError: no method matching MixtureModel(::Vector{Num}, ::Vector{Num})
Closest candidates are:
  MixtureModel(::Type{C}, ::AbstractArray) where C<:Distribution at C:\Users\metivier\.julia\packages\Distributions\Spcmv\src\mixtures\mixturemodel.jl:127
  MixtureModel(::Type{C}, ::AbstractArray, ::Vector{T}) where {C<:Distribution, T<:Real} at C:\Users\metivier\.julia\packages\Distributions\Spcmv\src\mixtures\mixturemodel.jl:144
  MixtureModel(::Vector{C}, ::VT) where {C<:Distribution, VT<:(AbstractVector{<:Real})} at C:\Users\metivier\.julia\packages\Distributions\Spcmv\src\mixtures\mixturemodel.jl:138
Stacktrace:
 [1] f(θ::Symbolics.Arr{Num, 1})
   @ Main c:\Users\metivier\Dropbox\PC (2)\Documents\Simulations\julia_mwe\ModelingToolkit_test_for_TrigMixture\mwe_modelingtoolkit_discourse.jl:4
 [2] ℓ(θ::Symbolics.Arr{Num, 1}, x::Symbolics.Arr{Num, 1})
   @ Main c:\Users\metivier\Dropbox\PC (2)\Documents\Simulations\julia_mwe\ModelingToolkit_test_for_TrigMixture\mwe_modelingtoolkit_discourse.jl:5
 [3] top-level scope
   @ c:\Users\metivier\Dropbox\PC (2)\Documents\Simulations\julia_mwe\ModelingToolkit_test_for_TrigMixture\mwe_modelingtoolkit_discourse.jl:25

It seems that Distributions.jl and ModelingToolKit.jl are not compatible in general, since there are type check, logsumexp with if statements, etc do not like Num.
I saw that DistributionsAD.jl was a good candidate to answer the issue. However, I was not able to understand how it works (could not find documentation).

Anyone with an idea how to implement that?

5 posts - 3 participants