One relatively new feature of Convex which I haven’t mentioned was introduced in 0.12.6: the “problem depot”. Convex.jl has a collection of optimization problems that it uses for end-to-end tests; the problem depot provides a way to easily use these test problems outside of Convex.jl’s own tests. These problems provide a means to check that Convex.jl is correctly modelling problems (which was their original purpose), but they also can be used to test that solvers are correctly solving them.
Since Convex.jl supports MathOptInterface (MOI), Convex 0.13 makes this suite of problems available for any MOI-compatible solver. I have setup a repository to test each open source (and non-matlab-based) MOI-compatible solver I could find: GitHub - ericphanson/ConvexTests.jl: Uses the Convex.jl Problem Depot to test various optimization solvers. This uses Github Actions to run all the tests in parallel and prints the results, along with timing information, in a readable manner in the documentation pages. Commerical and matlab-based solvers are only excluded because I don’t know how to run them on continuous-integration services, or if that’s possible.
The problem depot, and earlier forms thereof, was used for testing our wrapper of SDPA-GMP and its variants (SDPAFamily.jl), leading to the discovery of a quickly-fixed MOI correctness bug (Bug in SOCtoPSDBridge · Issue #838 · jump-dev/MathOptInterface.jl · GitHub; the only correctness bug on the tracker-- which itself is a testament to the high quality development of MOI), as well as the development of a presolve routine for SDPA-GMP, which allows the solver to pass 36 tests from the problem depot that it otherwise fails, although unfortunately at the cost of long presolve computation time for 3 problems it could already solve-- I am hoping to look into this at some point and speed up the presolve routine.
I think therefore the problem depot has already been a useful resource, and I hope the tooling in ConvexTests.jl makes it easily available to a wider audience. I hope also it can serve as a bit of a call-to-action, because many solvers fail at least some tests in the problem depot.