How to write a new gradient descent algorithm and benchmark against existing methods

Specific Domains Optimization (Mathematical)
1

I have an idea for a new gradient descent algorithm. I just want to see if it does better or worse than a standard momentum methods.

How would I go about making use of the existing benchmarking suites CUTEst.jl or OptimizationProblems.jl. Their manuals show how to run them on existing solvers, but how would I go about writing my own solution method. Cant seem to find anything in the JuMP or ADNLPModels manual. The code is somewhat elaborate in JSOSolvers.jl/src at main · JuliaSmoothOptimizers/JSOSolvers.jl · GitHub. Whats the bare minimum I need to run these test suites? Thanks!

2 Likes
2

HI @cdawg, thanks for the interest.

I’ve made these videos a few years ago: https://www.youtube.com/watch?v=DW9UAXDh77g (and parts 2 and 3).

And we also wrote this post: Creating a JSO-compliant solver

These might be a bit outdated, but most of it should work.

We also started GitHub - JuliaSmoothOptimizers/JSOSolverSkeleton.jl: JSO-compliant Solver templates last year, but it is due to an update.

Hope these help!

6 Likes
3

Hi @cdawg !

To add-on @abelsiqueira response.

Essentially, to run the benchmark you need a function that takes an AbstractNLPModel as input and return a GenericExecutionStats.

Once, you have this, you can adapt the benchmark:

to select the appropriate problems, and add your solver to the list.

Feel free to open an issue to one of these packages if you have trouble with the prototype of your solver or the benchmark scripts.

4 Likes
4

Your post Creating a JSO-compliant solver is a super helpful resource, thanks @abelsiqueira, and the hard work on these nice packages @tmigot ! It be great if this were in the docs of JSOSolvers.jl or OptimizationProblems.jl or both. I really appreciate your help. I will try this out. It may take a some days, but I will report back when i get things working or get stuck.

P.S. I don’t understand the solver skeleton package, will this make the process easier?

5

The motivation for the skeleton package was to have a Template to make a solver package. If you do
image
then you can personalize a copy of the skeleton package for your own use. It simplifies the process of making a package not necessarily coding the algorithm :slight_smile:

2 Likes