I am not sure if the Package Announcements category existed back when the previous version announcements were made about Optim.jl, so I am starting a new thread here.
Optim v1.8.0 is out as of yesterday. It is a feature release because @blegat has added MathOptInterace support (Introduction · MathOptInterface) thereby closing one of the oldest issues in Optim.jl. I also made the update because I found out that the docs were not running.
Lastly, I want to point something out. I saw some discussions where people had some ideas that Optim.jl may not be maintained, may be deprecated, and more. I just wanted to say, that that is definitely not the case. I have been busy with work at times and have not reviewed and responded to issues quickly. Actually, very slowly if at all. I have changed my Github settings and mail inbox settings to hopefully help highlight these so I don’t forget them or miss them. That said, it is always possible and welcomed to ping me on slack, and I’ll probably get to it rather quickly. Though, the fact that there are not new versions every month does not reflect that the package is dead, rather it probably means that it has met a relatively stable state.
For a very long time I have also been working on NLSolvers.jl that is supposed to replace Optim.jl internals at some point. It is tagged, and I am also happy to help with issues in that package whenever it may be needed. The package tries to avoid a lot of issues I had with Optim.jl and supports non-allocating versions of both equation solving and optimization. I will make a different post in the near future about the ideas and plans for that package.
Here at the end I also want to point out that there have been several unconstrained optimization, constrained optimization, nonlinear equation solving, nonlinear least squares solving packages in the ecosystem for a long time. That is good! I think they all fill various niches and needs. I use Optim.jl every day and so does our users, and it is adjusted and improved according to the needs that I and our product has. If users open issues and give feedback, their input will also help shape it. My needs are mostly met so that might be why there are not 20 new versions a year, and I think many other users find it useful as well. At least, more than 100 people and teams a year have been kind enough to cite their use in the past couple of years according to google scholar. It’s great to see that citing software has become normalized somewhat!
I’ve been using Optim.jl for years but I miss an Adam optimizer since it’s used so commonly in practice. Also the efficient fg! API requires always some boilerplate.
You are free to switch of course ! Any feature requests can go to the respective packages. NLSolvers.jl has Adam and AdaMax. I worked on that when I had more time, and the intention was to bring NLSolvers.jl code in as the next internal code for Optim.jl, but I have not gotten there yet
Optim v1.9.0 is out
A new feature release of Optim is out.
Based on @roflmaostc feedback I pulled in the Adam/AdaMax code from NLSolvers.jl, exported them as Adam and AdaMax, and added very brief mentions in the docs. Since these are fixed step length methods the user may have to set the step length themselves, but I didn’t make it a required keyword. Instead, I used the values from the paper.
Edit:
Optim v1.9.1 is out @roflmaostc found out that some operations in Adam and AdaMax were using scalar indexing and thus not CUDA compatible so a small bug fix was release as well
This looks very nice. It would be a make a great solver backend for my NLLSsolver.jl package. That package focuses more on problem definition (i.e. frontend), with features such as:
automatic computation of gradient and hessian (including sparse hessians)
support for robust kernels, including optimizing their parameters
support for manifold variables
However, it lacks support for:
a broad variety of solvers
constraints
So it seems that NLSolvers.jl offers solutions to that. In order to use it as the backend, NLSolvers.jl would need to support manifold variables, which I see is on your roadmap. I’d be happy to discuss it with you, when you’re at that stage.
Yes, manifolds are only included in a few places in the form of some basic scaffolding I am working on some fixes and documentation. I assume you would want to wait until that is out so you can better evaluate if it’s a good fit.
Did anyone notice a performance regression from v1.7 to v1.8 and v1.9? We have some scripts that are taking much longer to finish now after the update. We are not sure yet if the regression is coming from Optim.jl.
The optimization problem that we think was affected is this one:
We add box constraints with tight bounds sometimes (lower ~= upper).
If you’re tracking run-time hopefully you store a manifest as well such that you can see what packages were used? Maybe it’s better to make a separate thread for this discussion?
If you are interested, we could also check how to couple Optim.jl and ManifoldsBase.jl – we had some discussions quite a while back and concluded, that might be something nice as well. So if you are interested in that, let me know – I am happy to help.
It might not be easily doable in every solver, but in some probably.
Any feature requests can go to the respective packages. NLSolvers.jl has Adam and AdaMax. I worked on that when I had more time, and the intention was to bring NLSolvers.jl code in as the next internal code for Optim.jl, but I have not gotten there yet 
