Yes, I understood that, but thanks for making sure though.
Yes I too think that BOBYQA making not only progress but amazing progress while LBFGS didn’t suggests a gradient issue. It also suggests that the function shouldn’t be “too hard” to optimize because BOBYQA uses quadratic approximations and goes BANG right to the optimum, so I feel like this should be approximately a big normal distribution with probably a dense covariance matrix, lots of correlated parameters etc.
This is based on the maximum_a_posteriori interface in Turing, so underlying that it’s using Optimization.jl which specifies algorithms in terms of passing structs.
I can pass the LBFGS() struct, in which case I believe that Optimization.jl uses Fminbox with LBFGS inside that… so Fminbox is basically adding penalties and then running LBFGS and then adjusting the penalties in a loop I think.
It’s entirely possible it’s this Fminbox that’s the problem? I don’t know. But that wouldn’t explain why LD_LBFGS in NLopt doesn’t work. Though perhaps there it was not supporting box constraints… There are many things that are confusing to me.
One thing I may try is to fit with BOBYQA the full model with less constraints… find an optimum, perturb it a bit, and then run LBFGS from this perturbed optimum with NO constraints and see if it fits. perhaps Fminbox is the real problem and LBFGS would work fine if it were started close enough to the proper optimum.
I’ll do some experimentation, but a 12 hour marathon session yesterday means I probably will take today off. I’ll be back in the next few days with any info I can figure out though.