# How to write a new gradient descent algorithm and benchmark against existing methods

**URL:** <https://discourse.julialang.org/t/how-to-write-a-new-gradient-descent-algorithm-and-benchmark-against-existing-methods/119869>\
**Category:** Optimization (Mathematical)\
**Tags:** question\
**Created:** [September 25, 2024, 2:21pm UTC](https://discourse.julialang.org/t/how-to-write-a-new-gradient-descent-algorithm-and-benchmark-against-existing-methods/119869 "2024-09-25T14:21:41Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cdawg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cdawg/32/9811_2.png) [@cdawg](https://discourse.julialang.org/u/cdawg)\
**Post date:** [September 25, 2024, 2:21pm UTC](https://discourse.julialang.org/t/how-to-write-a-new-gradient-descent-algorithm-and-benchmark-against-existing-methods/119869/1 "2024-09-25T14:21:41Z")

</div>

I have an idea for a new gradient descent algorithm. I just want to see if it does better or worse than a standard momentum methods.

How would I go about making use of the existing benchmarking suites CUTEst.jl or [OptimizationProblems.jl](https://github.com/JuliaSmoothOptimizers/OptimizationProblems.jl). Their manuals show how to run them on existing solvers, but how would I go about writing my own solution method. Cant seem to find anything in the JuMP or [ADNLPModels](https://github.com/JuliaSmoothOptimizers/ADNLPModels.jl) manual. The code is somewhat elaborate in [JSOSolvers.jl/src at main · JuliaSmoothOptimizers/JSOSolvers.jl · GitHub](https://github.com/JuliaSmoothOptimizers/JSOSolvers.jl/tree/main/src). Whats the bare minimum I need to run these test suites? Thanks!

---

<div class="post-metadata">

**Author:** ![abelsiqueira](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/abelsiqueira/32/47269_2.png) [@abelsiqueira](https://discourse.julialang.org/u/abelsiqueira)\
**Post date:** [September 25, 2024, 2:46pm UTC](https://discourse.julialang.org/t/how-to-write-a-new-gradient-descent-algorithm-and-benchmark-against-existing-methods/119869/2 "2024-09-25T14:46:59Z")

</div>

HI @cdawg, thanks for the interest.

I’ve made these videos a few years ago: [https://www.youtube.com/watch?v=DW9UAXDh77g](https://www.youtube.com/watch?v=DW9UAXDh77g) (and parts 2 and 3).

And we also wrote this post: [Creating a JSO-compliant solver](https://jso.dev/tutorials/creating-a-jso-compliant-solver/)

These might be a bit outdated, but most of it should work.

We also started [GitHub - JuliaSmoothOptimizers/JSOSolverSkeleton.jl: JSO-compliant Solver templates](https://github.com/JuliaSmoothOptimizers/JSOSolverSkeleton.jl) last year, but it is due to an update.

Hope these help!

---

<div class="post-metadata">

**Author:** ![tmigot](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tmigot/32/23914_2.png) [@tmigot](https://discourse.julialang.org/u/tmigot)\
**Post date:** [September 25, 2024, 3:11pm UTC](https://discourse.julialang.org/t/how-to-write-a-new-gradient-descent-algorithm-and-benchmark-against-existing-methods/119869/3 "2024-09-25T15:11:52Z")

</div>

Hi @cdawg !

To add-on @abelsiqueira response.

Essentially, to run the benchmark you need a function that takes an AbstractNLPModel as input and return a GenericExecutionStats.

Once, you have this, you can adapt the benchmark:

> **[Benchmark · OptimizationProblems.jl](https://jso.dev/OptimizationProblems.jl/dev/benchmark/)**
>
> Documentation for OptimizationProblems.jl.

to select the appropriate problems, and add your solver to the list.

Feel free to open an issue to one of these packages if you have trouble with the prototype of your solver or the benchmark scripts.

---

<div class="post-metadata">

**Author:** ![cdawg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cdawg/32/9811_2.png) [@cdawg](https://discourse.julialang.org/u/cdawg)\
**Post date:** [September 26, 2024, 7:15am UTC](https://discourse.julialang.org/t/how-to-write-a-new-gradient-descent-algorithm-and-benchmark-against-existing-methods/119869/4 "2024-09-26T07:15:38Z")

</div>

Your post [Creating a JSO-compliant solver](https://jso.dev/tutorials/creating-a-jso-compliant-solver/) is a super helpful resource, thanks @abelsiqueira, and the hard work on these nice packages @tmigot ! It be great if this were in the docs of JSOSolvers.jl or OptimizationProblems.jl or both. I really appreciate your help. I will try this out. It may take a some days, but I will report back when i get things working or get stuck.

P.S. I don’t understand the solver skeleton package, will this make the process easier?

---

<div class="post-metadata">

**Author:** ![tmigot](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tmigot/32/23914_2.png) [@tmigot](https://discourse.julialang.org/u/tmigot)\
**Post date:** [September 26, 2024, 11:06am UTC](https://discourse.julialang.org/t/how-to-write-a-new-gradient-descent-algorithm-and-benchmark-against-existing-methods/119869/5 "2024-09-26T11:06:07Z")

</div>

The motivation for the skeleton package was to have a Template to make a solver package. If you do  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/4/4/4412b99adc356f5a1fbf174d568dc5093b00b4b2.png)  
then you can personalize a copy of the skeleton package for your own use. It simplifies the process of making a package not necessarily coding the algorithm 🙂