I have some empirical values for the cdf of a distribution which should be close to Normal. These are for x = 1, 2, … , 13. They are as follows: [5.2909066017292616e-17, 2.998035847356917e-15, 2.893916126231429e-12, 2.893916126231429e-12, 5.984024927027287e-11, 9.828060393355409e-10, 1.306578361293…

This seems to work: μ_hat, σ_hat = optimize(cdfcost, [-100.0, 0.01], [100.0, 10.0], [0.0, 1.0]).minimizer plot(0.1:0.1:13, [cdf(Normal(μ_hat, σ_hat), x) for x in 0.1:0.1:13]) scatter!(1:13, empiricalcdf)

[image] nilshg: μ_hat, σ_hat = optimize(cdfcost, [-100.0, 0.01], [100.0, 10.0], [0.0, 1.0]).minimizer plot(0.1:0.1:13, [cdf(Normal(μ_hat, σ_hat), x) for x in 0.1:0.1:13]) scatter!(1:13, empiricalcdf) Is the only difference that you have added a box constraint?

Well it also uses a different algorithm and starting point - it looks like this problem is not well behaved? Consider (just picked the first few examples from the docs): μ_hat, σ_hat = optimize(cdfcost, [-100.0, 0.01], [100.0, 10.0], [0.0, 1.0]).minimizer μ_hat2, σ_hat2 = optimize(cdfcost, [0.0, 1.…

[image] nilshg: optimize(cdfcost, [0.0, 1.0], BFGS()).minimizer All that we are doing is varying the mean and variance of a normal. It feels like the optimization should be well behaved.

Not really. The small probabilities don’t identify anything strongly (0 is a very good fit, producing a small distance metric), so you get iso-cost lines. When in doubt, plot. Eg using Distributions, PGFPlotsX empiricalcdf = [5.2909066017292616e-17, 2.998035847356917e-15, 2.893916126231429e-12, …

I’m going to just second what @Tamas_Papp has already said. The data are too far into the tail to really be informative. Based on your other-other post, you using a skew normal, that is actually only going to make the problem worse because it is actually a more flexible distribution. That means that…

To be fair, I didn’t really think it would be straightforward. But I was hoping that with the very high precision of BigFloat optimization might work. I have in mind that 3 points (even from the tail if one has enough precision) is enough to uniquely define a normal distribution. Perhaps 4 is suf…

I think your conjectures would be correct if indeed the points came from such distributions. But with noisy data, anything goes.

It’s in fact worse than that. My data is in fact not noisy (I have precise rational values for the densities which I convert to BigFloat). It just doesn’t come from precisely a normal distribution. It will be close to a normal and even closer to a skew normal I suspect.

How to fit a normal distribution to some points in the tail

Specific Domains

Tamas_Papp February 27, 2020, 5:38pm 7

FWIW, the likelihood I wrote for what I imagine is the same problem should work for the truncated data.

Topic		Replies	Views
Optimization giving wildly different answers depending on box constraints Optimization (Mathematical)	2	659	February 27, 2020
Need advice when data almost (but not quite) fit log normal dist Statistics distributions	7	1031	January 13, 2021
Fitting a Distribution to existing data Statistics	16	6668	January 5, 2017
Fitting a truncated distribution Specific Domains fit , distributions	0	78	January 22, 2025
How to fit a normal approximation to data in Julia Statistics	21	5029	January 8, 2021

How to fit a normal distribution to some points in the tail

Related topics