Parameters Optimization with Optim

General Usage
1

Hi Everyone,
I have a model that is regulated by a set of parameters. I am trying to choose the parameters so that the model matches some moments that I get from the data.

This is my code:

lower_bounds = [0.0, 1e-2,   1e-2, 1e-2, 0]
upper_bounds = [1, 1,   1,  2, 0.9999]


# 5. Optimization with constraints and max iterations
 using Optim


# Define optimization options with maximum iterations and trace display
opt_options = Optim.Options(
     f_tol = 1e-3,  
     g_tol=1e-3,          
     iterations = 1000,          
     show_trace = true,        # Display progress at each iteration
     store_trace = true,       # Keep a record of parameters and objective history
     extended_trace = true   # Display additional information like gradient norm
 )

params=parameters
    res = optimize(
        parameters -> objective(parameters, data_moments, W),
        lower_bounds,
        upper_bounds,
        params,  # Use the current set of initial parameters
        Fminbox(NelderMead()),  
        opt_options
    )

Where the function objective solves the model and compute the implied moments from the model and the distance via sum of weighted squared errors.

This is the outcome that I am receiving:

Parameters: [0.536, 0.809, 0.294, 0.8995, 0.94] Moments: [0.52889671328668, 0.38916868, 0.17606923, 0.32529774, 0.43348429] Error: 0.0006003618478701606
Err at initial guess 0.0006003618478701606
Parameters: [0.536, 0.809, 0.294, 0.8995, 0.94] Moments: [0.5288967132866799, 0.38943109, 0.17588286, 0.32560986, 0.43185353] Error: 0.0005998821941473227
Parameters: [0.8290000000000001, 0.809, 0.294, 0.8995, 0.94] Moments: [0.5252191775772682, 0.38218966, 0.19402218, 0.22010095, 0.4038389] Error: 0.0014946071162564058
Parameters: [0.536, 0.809, 0.46599999999999997, 0.8995, 0.94] Moments: [0.5233505501989021, 0.64591044, 0.18129237, 0.35279214, 0.46952623] Error: 0.024091420780591264
Parameters: [0.536, 0.809, 0.294, 1.37425, 0.94] Moments: [0.5249256301176558, 0.39822295, 0.19268225, 0.33760324, 0.43480611] Error: 0.0003246306002237922
Parameters: [0.5360060554544525, 0.809, 0.294, 0.8995, 0.94] Moments: [0.5288966905746657, 0.38932863, 0.17603225, 0.32645905, 0.43323335] Error: 0.0005988598443701483
Parameters: [0.5359939445455476, 0.809, 0.294, 0.8995, 0.94] Moments: [0.5288967818925795, 0.38952717, 0.17595945, 0.32588565, 0.43420699] Error: 0.0005977599843214626
Parameters: [0.536, 0.8090060554544525, 0.294, 0.8995, 0.94] Moments: [0.5288971816897843, 0.38948107, 0.17608127, 0.32551727, 0.43259948] Error: 0.0005967244661222952
Parameters: [0.536, 0.8089939445455476, 0.294, 0.8995, 0.94] Moments: [0.528896252637118, 0.38940319, 0.17597665, 0.32672238, 0.43200535] Error: 0.0005987646288916327
Parameters: [0.536, 0.809, 0.29400605545445235, 0.8995, 0.94] Moments: [0.5288966047283064, 0.3894273, 0.17600751, 0.32676721, 0.43264592] Error: 0.0005980919534752674
Parameters: [0.536, 0.809, 0.2939939445455476, 0.8995, 0.94] Moments: [0.5288968507034159, 0.38945293, 0.17593241, 0.32615566, 0.43157122] Error: 0.0005988756950694417
Parameters: [0.536, 0.809, 0.294, 0.8995060554544524, 0.94] Moments: [0.5288966440422649, 0.38928875, 0.17598616, 0.32581824, 0.43137595] Error: 0.0006000095168527948
Parameters: [0.536, 0.809, 0.294, 0.8994939445455475, 0.94] Moments: [0.5288967889254054, 0.38942745, 0.17596379, 0.32503468, 0.4330371] Error: 0.0005990168453222712
Parameters: [0.536, 0.809, 0.294, 0.8995, 0.9400060554544524] Moments: [0.5288988226793248, 0.3895129, 0.17608249, 0.3260892, 0.43346983] Error: 0.000596252870306835
Parameters: [0.536, 0.809, 0.294, 0.8995, 0.9399939445455475] Moments: [0.528894604761689, 0.38934618, 0.17594257, 0.32708284, 0.43504655] Error: 0.00059982219984397
Fminbox
-------
Initial mu = 1.98846e-5

Fminbox iteration 1
-------------------
Calling inner optimizer with mu = 1.98846e-5

(numbers below include barrier contribution)
Iter     Function value    √(Σ(yᵢ-ȳ)²)/n 
------   --------------    --------------
     0     3.246306e-04              NaN
 * time: 0.031883955001831055
 * step_type: initial
 * centroid: [0.5946000000000001, 0.8949, 0.3284, 0.9944500000000001, 0.94]
Parameters: [0.5946000000000001, 0.8949, 0.3284, 0.9944500000000001, 0.94] Moments: [0.5339515807509395, 0.47850657, 0.15678309, 0.32810655, 0.39227873] Error: 0.0031405109961025078

Exiting inner optimizer with x = [0.536, 1.2385, 0.294, 0.8995, 0.94]
Current distance to box: -0.2385
Decreasing barrier term μ.

The,. after this I receive an error because 1.2385 is outside the possible values for the admissible parameters.

My questions are:

  1. Why is it exiting with 1.2385? In the above example the set of parameters that minimizes the function is:

Parameters: [0.536, 0.809, 0.294, 1.37425, 0.94] Moments: [0.5249256301176558, 0.39822295, 0.19268225, 0.33760324, 0.43480611] Error: 0.0003246306002237922

Why is the function jumping to second parameter equal to 1.2385?

  1. I thought of using NelderMead because the function is not smooth and I wanted to avoid a gradient-based method. Let me know if you have any suggestions or if you would change anything.

Thank you so much for your help, is really appreciated :slight_smile:

2
  1. Did you try other gradient-free methods?
  2. Did you try reducing the step size of Nelder-Mead?
3

Thank you for your answer. What would you suggest I use instead? I thought Nelder-Mead was the best candidate for my case.

I would also like to understand the root cause of the problem. With Nelder-Mead, I thought the function evaluates the objective at each vertex of the simplex and then applies transformations using combinations of previous and new parameter values. In this case, no such combination seems to justify the jump to 1.2, so I’m really confused about what’s happening here.

4

But how big is the simplex?
You should be able to adjust the initial simplex and control later steps by changing the parameters.

Also I would try all other algorithms available by Optim.jl, even if the gradient is required.
If you don’t pass the gradient then Optim.jl should approximate it.

5

Can you provide a complete runnable example, including function definition?