Optimization.jl - Define the objective function and its gradient in the same function

Specific Domains Optimization (Mathematical)
1

Hi all, I am working on an optimization problem involving Complex Gaussian Processes and Kalman Filter. To optimize the hyperparameters of the kernels, I use Optimization.jl. Because I have calculated the analytical gradients, I can use gradient-based methods.

I define the OptimizationFunction as:

OptimizationFunction(objfun!, grad = objfun_gradient!)

It works, but objfun! and objfun_gradient! share a lot of computations. So, it would be ideal if I can define the objective function and its gradient in the same function in order to save computation time and resources.

My question is: Is it possible to do this with Optimization.jl? My understanding of the documentation is that it is not possible, but a clever solution may exist.

Thanks!

2

I think individual solvers like Optim.jl may support it, but it doesn’t seem to be planned for the interface library: see Is there an equivalent to Optim.jl's only_fg! ? · Issue #1101 · SciML/Optimization.jl · GitHub

3

@gdalle Thank you for your accurate answer (as usual).

4

I think you can hack it together using a closure or a functor:

mutable struct ComputeCache
    arg::Vector{Float64}
    func::Float64
    grad::Vector{Float64}
end

struct ObjFunc{F}<:Function
    cache::ComputeCache
    gf::F
end

struct GradObjFunc{F}<:Function
    cache::ComputeCache
    gf::F
end

function (fn::ObjFunc)(x)
    if x == fn.cache.arg
        return fn.cache.func
    else
        g, f = fn.gf(fn.cache.grad, x)
        fn.cache.arg .= x
        fn.cache.func = f
        return f
    end
end

function (fn::GradObjFunc)(grad, x)
    if x != fn.cache.arg
        g, f = fn.gf(fn.cache.grad, x)
        fn.cache.arg .= x
        fn.cache.func = f
    end
    grad .= fn.cache.grad
    return grad
end

cache = ComputeCache(...)

OptimizationFunction(ObjFunc(cache, gf!); grad=GradObjFunc(cache, gf!))