Re-architecting NLLSsolver.jl into modular packages (NLLSModel.jl, SecondOrderOptimization.jl, and friends)

I’ve recently started registering several new packages, and have been asked on the registry PRs what they’re for. That deserves a fuller answer than fits in a PR comment, so here it is, along with an invitation to ask questions.

Background

I wrote and maintain NLLSsolver.jl, a non-linear least squares package with manifold variables and robust kernels. It is fast, but it is also niche and monolithic: you have to buy into its whole way of describing a problem before you get anything out of it. I think that learning cost (no doubt compounded by a scarcity of documentation) is the main reason relatively few people use it.

For that reason, I’ve decided to split it up into smaller, complementary but also more generally compatible (and properly documented) packages.

The idea

Second-order optimization is quite modular. There are four largely independent parts:

  1. A cost function (the model): something that returns a cost, and a gradient and Hessian, or residuals and a Jacobian.
  2. An optimization loop: evaluate, propose a step, accept or reject, test for convergence.
  3. Iterators that decide what happens inside the loop: Newton, Levenberg-Marquardt, dogleg trust region, etc.
  4. Linear solvers, which depend on the problem structure (static, dense, sparse, block-sparse with eliminable variables), and for which highly performant packages already exist.

In NLLSsolver these were tangled together. I’m separating them, so that each part can be used on its own with other packages, and so that each part can be swapped for one from another ecosystem.

The packages

Package Role Status
OptimizationVariableInterface.jl How an optimizer updates a variable (retractions, manifolds, custom types). Announced here. Registered
NLLSModel.jl Part 1. Factor-graph modelling of least squares costs: autodiff or analytic Jacobians, robust kernels, multi-threaded assembly. In development
SecondOrderOptimization.jl Parts 2 and 3. The loop and the iterators, for any smooth scalar objective, not only least squares. Registration paused
DampedLinearSolve.jl Part 4. A small contract for the damped solves these iterators need, implemented for static, dense and sparse systems, with Krylov.jl and Pardiso.jl via extensions. Awaiting registration
BlockLinearSystems.jl Part 4. Block-structured Hessian and Jacobian storage, with Schur complement and QR elimination. Registered

The model and the optimizer have no hard dependency on each other. In practice this means:

  • NLLSModel.jl with other optimizers. A model exposes plain f / fdf! functions, and wraps as an AbstractNLPModel, so it can be solved with JSOSolvers.jl, Ipopt, Optim.jl, etc.
  • SecondOrderOptimization.jl with other models. It needs only a cost with a gradient and Hessian, or a residual and Jacobian. It knows nothing about NLLSModel.
  • Other variable types in either. Anything implementing OVI works, including Manifolds.jl manifolds via the ManifoldsBase extension. Other optimizers could adopt OVI to gain the same.
  • Other linear solvers. The solve sits behind a contract, so new system types and solvers can be added from outside the optimizer.

NLLSsolver.jl itself will be deprecated. In the meantime it will become a thin wrapper that re-exports the new packages, so existing code keeps working.

What this adds to the landscape

Carried over from NLLSsolver: manifold variables, robust kernels, and speed. New in the split:

  • Allocation-free (stack-only) optimization of small, statically sized problems.
  • Least squares solved from the residual and Jacobian directly, without forming JᵀJ, including with robust kernels.
  • Elimination of variables on both the normal-equation (Schur) and Jacobian (QR) paths, for large structured problems.

I’m also building SmoothOptimizerBenchmark.jl, which compares these packages against LeastSquaresOptim, Optim, JSOSolvers, Ipopt, MadNLP, NonlinearSolve, CaNNOLeS and Manopt across problem types and sizes, timing model evaluation and optimizer separately. Once that is up and running (any day now), I’ll share a link to the results in this thread. If your package is in that list, I’d welcome a check that I’m using it fairly.

Current state

These are early (pre v1.0) releases, and the APIs may change quite often. Breaking changes will trigger a new minor release (prior to v1).

A note on process, since it was raised: a lot of the code in these packages is authored by AI. I haven’t written a two-line prompt and left it to do everything, though. I work in a tight iterative loop with it: the architecture, algorithms and API decisions are mine, I give a lot of input at each step, and I review a lot of what comes back. I’m happy to take questions on this too.

Questions welcome

I’d particularly like to hear:

  • whether the boundaries between the packages are in the right places;
  • from maintainers of optimizers or modelling packages who might want to adopt OVI, or to plug into either side of the model/optimizer split;
  • where any of this duplicates something that already exists and I should be building on that instead.

Thanks,
Oliver

1 Like