I do have a loss function that :
- Is expressed as \lVert \mathbf y - \mathbf A'\mathbf p(\mathbf x)\rVert_2^2, where \mathbf y is a simple vector, \mathbf A is a simple matrix (lower-triangular), and p_1,...,p_m are nasty polynomials.
- Polynomials p_1,...,p_m are computed through a recursive implementation, say a function
p(m,x)that computes all of them at once and which should be considered a black box (the number of coefficients of the polynomials are in the millions if we try to express them explicitely…).
- The loss is written in pure julia and automatic differentiation can pass through
- But the very polynomial nature of the loss gives it a huge amount of local minimas, and any gradient descent (e.g.
Optim.LBFGS()) gets stuck on the nearest local minimum.
For all these reasons, I am currently optimizing through
Optim.ParticleSwarm(), which is a global algorithm for non-convex non-differentiable functions. It works well but it’s quite slow.
Is there somewhere a global optimization algorithm that is more adapted to problems where:
- The loss function is clearly non-convex with a lot of local minima.
- But it is ‘smooth’ and automatic differentiation can pass through, and therefore local gradient descent are possibles.
It is required that the optimisation routines are implemented in pure julia as i use BigFloats. Furthermore if linear equality and bound contraints are posible it’s a bonus