Optim.jl wrongly reporting success

Specific Domains Optimization (Mathematical)
1

I’m solving for the MLE.

using Distributions, Random, Optim;
DGP_True = LogNormal(17,7);
Random.seed!(123);
const d_train = rand(DGP_True, 1_000)
const d_test  = rand(DGP_True, 1_000)

# LogNormal
function loglike(θ)
    dist          = LogNormal(θ[1],θ[2])
    terms         = logpdf.(dist,d_train)
    loglikelihood = sum(terms)
    return -loglikelihood #Min(-LogLik) =Max(LogLik)
end
#
optimum = optimize(loglike,ones(2))
Optim.minimizer(optimum)
Distributions.fit_mle(LogNormal,d_train)

# Optim.jl &  Distributions.jl solutions are very close.


# Normal 
function loglike(θ)
    dist          = Normal(θ[1],θ[2])
    terms         = logpdf.(dist,d_train)
    loglikelihood = sum(terms)
    return -loglikelihood #Min(-LogLik) =Max(LogLik)
end
#
optimum = optimize(loglike,ones(2))
Optim.minimizer(optimum)
Distributions.fit_mle(Normal,d_train)

# Optim.jl reports "sucess" BUT
# Optim.jl &  Distributions.jl solutions are very different!

A second example:

using Distributions, Random, Optim;
DGP_True = LogNormal(0,1);
Random.seed!(123);
const d_train = rand(DGP_True, 1_000);
const d_test  = rand(DGP_True, 1_000);

D1=Distributions.fit_mle(Laplace, d_train)

function loglike(θ)
    dist          = Laplace(θ[1],θ[2])
    terms         = logpdf.(dist,d_train)
    loglikelihood = sum(terms)
    return -loglikelihood #Min(-LogLik) =Max(LogLik)
end
sol=optimize(loglike,ones(2),Optim.Options(iterations=10^4))
D2=Optim.minimizer(sol)

The estimate for the first parameter is very similar:
μ=1.0112216426398657 vs Optim’s 1.0116066436206388
The estimate for the 2nd parameter is different:
θ=0.9005083163386765 vs Optim’s 1.1872092883101257

My main concern is not that Optim.jl is not able to solve this, it is that it is wrongly reporting “success”.

I created an issue, but it was closed for some reason…

2

Optimization methods report success when they reach a gradient which is small enough, within the tolerance set by the user. If the gradient is correct (is it?) and within the tolerance, that is expected. That does not mean that one is always satisfied with the result, though.

3

It was closed because your code has some numerical issues that seem to post challenges for any numerical optimization technique, no? Do you have an example of the code behaving properly in another library that Optim could try to emulate?

4

I closed it because it didn’t appear to contain anything actionable, and the other people in the thread had taken time to show you that it’s probably just a case of an ill-conditioned problem. If it helps, Optim won’t use the word “success” in the future.

2 Likes
5

JuMP uses “locally solved” for this, which has reduced confusion.