# Optim.jl - get the hessian matrix from maximum likelihood estimation

**URL:** <https://discourse.julialang.org/t/optim-jl-get-the-hessian-matrix-from-maximum-likelihood-estimation/70615>\
**Category:** Optimization (Mathematical)\
**Tags:** question\
**Created:** [October 29, 2021, 11:44am UTC](https://discourse.julialang.org/t/optim-jl-get-the-hessian-matrix-from-maximum-likelihood-estimation/70615 "2021-10-29T11:44:04Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Strange\_Xue](https://avatars.discourse-cdn.com/v4/letter/s/e9a140/32.png) [@Strange\_Xue](https://discourse.julialang.org/u/Strange_Xue)\
**Post date:** [October 29, 2021, 11:44am UTC](https://discourse.julialang.org/t/optim-jl-get-the-hessian-matrix-from-maximum-likelihood-estimation/70615/1 "2021-10-29T11:44:04Z")

</div>

There exists an example at Optim.jl’s [tutorial](https://julianlsolvers.github.io/Optim.jl/stable/#examples/generated/maxlikenlm/), I copy the code here.

```julia
using Optim, NLSolversBase, Random
using LinearAlgebra: diag
Random.seed!(0); # Fix random seed generator for reproducibility

n = 500 # Number of observations
nvar = 2 # Number of variables
β = ones(nvar) * 3.0 # True coefficients
x = [ones(n) randn(n, nvar - 1)] # X matrix of explanatory variables plus constant
ε = randn(n) * 0.5 # Error variance
y = x * β + ε; # Generate Data

function Log_Likelihood(X, Y, β, log_σ)
    σ = exp(log_σ)
    llike = -n/2*log(2π) - n/2* log(σ^2) - (sum((Y - X * β).^2) / (2σ^2))
    llike = -llike
end

func = TwiceDifferentiable(vars -> Log_Likelihood(x, y, vars[1:nvar], vars[nvar + 1]),
                           ones(nvar+1); autodiff=:forward);

opt = optimize(func, ones(nvar+1))

parameters = Optim.minimizer(opt)

parameters[nvar+1] = exp(parameters[nvar+1])

numerical_hessian = hessian!(func,parameters)

var_cov_matrix = inv(numerical_hessian)

β = parameters[1:nvar]

temp = diag(var_cov_matrix)
temp1 = temp[1:nvar]

t_stats = β./sqrt.(temp1)

# This file was generated using Literate.jl, https://github.com/fredrikekre/Literate.jl

```

as you see, `Log_Likelihood` function accepts log\_\sigma as one of parameters, so the estimated parameters should also be log\_\sigma, however, if you change log\_\sigma to \sigma by `parameters[nvar+1] = exp(parameters[nvar+1])`, then the hessian matrix is not right, as `Log_Likelihood` 's input is log\_\sigma, am I right?

I can’t figure out why to take exponential to log\_\sigma before calculating hessian matrix.

---

<div class="post-metadata">

**Author:** ![PharmCat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pharmcat/32/6953_2.png) [@PharmCat](https://discourse.julialang.org/u/PharmCat)\
**Post date:** [October 29, 2021, 11:01pm UTC](https://discourse.julialang.org/t/optim-jl-get-the-hessian-matrix-from-maximum-likelihood-estimation/70615/2 "2021-10-29T23:01:13Z")

</div>

I think you are right.  
Try to make 2 functions:

```julia
function Log_Likelihood(X, Y, β, σ)
    llike = -n/2*log(2π) - n/2* log(σ^2) - (sum((Y - X * β).^2) / (2σ^2))
    llike = -llike
end
function Log_Likelihood_opt(X, Y, β, log_σ)
    σ = exp(log_σ)
    Log_Likelihood(X, Y, β, σ)
end

```

Optimize second, and find Hessian with first.

---

<div class="post-metadata">

**Author:** ![mcreel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcreel/32/30088_2.png) [@mcreel](https://discourse.julialang.org/u/mcreel)\
**Post date:** [October 30, 2021, 7:03am UTC](https://discourse.julialang.org/t/optim-jl-get-the-hessian-matrix-from-maximum-likelihood-estimation/70615/3 "2021-10-30T07:03:27Z")

</div>

I think you are correct that the example is in error. It would be good to raise an issue at Optim.jl to call attention to the problem. The solution of using two functions is reasonable, but another way would be to use the [delta method](https://github.com/mcreel/Econometrics/blob/master/Examples/Restrictions/ExampleDeltaMethod.jl), which is often employed when computing transformations of estimated parameters.

---

<div class="post-metadata">

**Author:** ![mcreel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcreel/32/30088_2.png) [@mcreel](https://discourse.julialang.org/u/mcreel)\
**Post date:** [October 30, 2021, 7:46am UTC](https://discourse.julialang.org/t/optim-jl-get-the-hessian-matrix-from-maximum-likelihood-estimation/70615/4 "2021-10-30T07:46:21Z")

</div>

Actually, the Hessian matrix for the linear regression model with normality is block diagonal, see [https://www.cemfi.es/~arellano/Likelihood.pdf](https://www.cemfi.es/~arellano/Likelihood.pdf), page 7, for example. So, the confusion with the standard error of sigma will not affect the estimated standard errors of the coefficients, which is what the example focuses on. The t- statistics at the end of the example are correct, I believe. However, if one were to look at the estimated standard error of sigma, it would be incorrect.

---

<div class="post-metadata">

**Author:** ![mcreel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcreel/32/30088_2.png) [@mcreel](https://discourse.julialang.org/u/mcreel)\
**Post date:** [October 30, 2021, 9:45am UTC](https://discourse.julialang.org/t/optim-jl-get-the-hessian-matrix-from-maximum-likelihood-estimation/70615/5 "2021-10-30T09:45:36Z")

</div>

Final note: the t-statistics are not correct, as can be verified by computing the OLS estimator. This is because the Hessian is computed using the estimate of sigma, rather than log sigma. I have submitted a PR to fix this.

---

<div class="post-metadata">

**Author:** ![laurar1891](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/laurar1891/32/38443_2.png) [@laurar1891](https://discourse.julialang.org/u/laurar1891)\
**Post date:** [August 3, 2022, 7:07pm UTC](https://discourse.julialang.org/t/optim-jl-get-the-hessian-matrix-from-maximum-likelihood-estimation/70615/6 "2022-08-03T19:07:43Z")

</div>

Do you know if this has been solved now?

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [August 3, 2022, 7:13pm UTC](https://discourse.julialang.org/t/optim-jl-get-the-hessian-matrix-from-maximum-likelihood-estimation/70615/7 "2022-08-03T19:13:11Z")

</div>

See [fix for ML example by mcreel · Pull Request #956 · JuliaNLSolvers/Optim.jl · GitHub](https://github.com/JuliaNLSolvers/Optim.jl/pull/956) integrated into [Optim.jl 1.5](https://github.com/JuliaNLSolvers/Optim.jl/commit/fff90022e27e5b7e1449e0ceb39318ccd57b4af4).