I can partially comment on your questions:
- Analytical derivative is more efficient than auto-diff AFAIK.
- You may want to try different line search techniques from Home · LineSearches.jl much like Optim line search · LineSearches.jl. A line search is what it sounds like, searching for a good (ideally the best) solution on a line. Some line search techniques spend more time than others finding a solution to move to, but some times it can be better to just move to some good solution then find a new search direction to do another line search on. So it’s hard to say which one will be best for you without actually trying it. The line search techniques also have parameters that can make them more or less desperate to find a better solution so you have quite the parameter space to explore.
- You may also want to try other optimizers (search direction generators), not just L-BFGS, I found the ConjugateGradient to be very often very fast and robust for some of my problems.
- If you can find the Hessian of your objective function or some approximation of it, even just the diagonal, you may also want to use that approximate Hessian as a preconditioner in L-BFGS or ConjugateGradient which take a preconditioner Optim.jl. This will increase the cost of computing a search direction but it will improve its quality so you can benefit from spending more time (or function evaluations) searching along this expensively computed search direction, which may get you closer to the local optimal solution.
At the end of the day, only experimenting will show you the right path, I am only giving general advice.