Does automatic differentiation work for function with discrete argument?

[zxjroger]

I am trying to find the minimizer \theta of the following nonlinear and non-differentiable function: g(H(X | \theta),X). Here, X is known and imported from dataset; \theta=(\theta_1,\theta_2,\theta_3) are the parameters I want to solve for; H is discrete and therefore makes g(\cdot) non-differentiable with respect to \theta. There is only one simple constraint such that the first parameter \theta_1 is positive.

Since the argument H is discrete, I wonder if I can still use automatic differentiation in Ipopt for this optimization problem? Thank you.

[rdeits]

Well, yes and no. Automatic differentiation can handle discrete functions, but all it will do is (correctly) tell you that the derivative is zero:

julia> using ForwardDiff
[ Info: Precompiling ForwardDiff [f6369f11-7733-5829-9624-2563aa707210]
funct     
julia> function f(x)
         if x > 0
           return 1
         else
           return 2
         end
       end
f (generic function with 1 method)

julia> ForwardDiff.derivative(f, 1.0)
0

So using autodiff with an optimization method that expects a gradient is unlikely to work well. Perhaps one of the non-gradient-based algorithms from NLOpt would be more appropriate here? NLopt Algorithms - NLopt Documentation

[zxjroger]

I think my function is different from this. The function g(\cdot) is kind of smooth, although argument H is discrete. In your example, function f(\cdot) is discrete but argument x is smooth.

[Tamas_Papp]

@zxjroger

The function g(⋅)g(\cdot) is kind of smooth, although argument HH is discrete.

But the function you are dealing with is f(\theta) = g(H(X \mid \theta), X) which in general will not be continuous(ly differentiable) if H isn’t, even if g is.

Depending on how bad things are, stochastic (eg simulated annealing) or robust deterministic derivative-free optimization methods are used in these cases. See

Latest recommendations for global optimization

What are people’s recommendations for a global optimizer? I’m trying to do simulated method of moments (choosing a set of parameters for an economic model to minimise the distance between model predictions and data) and have so far been using blackboxoptim. I find it somewhat unreliable: running exactly the same code on the same data I find sometimes the optimizer gives me a parameter choice that provides a close fit to the data, but sometimes it just seems to get stuck without converging). Any …

Global optimization: Simulated Method of Moments

I work with non-linear models that need to be calibrated to match data moments. The setup is simple. I have a function that takes a set of parameters as input (for example, a vector of floats), solves the model, and returns a measure of the distance between the model-generated moments and the data moments. Typically there are more moments than parameters. Hence, I use some simple weighting function to summarize the differences between model and data moments into one float. The objective is to m…

[zxjroger]

Thanks for your comments. I have read these two posts previously. My problem is also simulated method of moments. I was testing blackbox and showed these two posts to my friend. But he still wants me to test Ipopt + user-defined numerical derivatives in Julia.

BTW, do you happen to know this new global optimizer named TikTak, which is developed by Fatih Guvenen?

[odow]

See Nonlinear Modeling · JuMP. You can pass a function that evaluates the gradients.

[zxjroger]

I do not think it will apply to my case. Basically, it only supports univariate function. I am using MathOptInterface instead and call Ipopt with self-defined derivatives.

[Tamas_Papp]

@zxjroger

this new global optimizer named TikTak

I implemented a Julia version of it in

GitHub - tpapp/MultistartOptimization.jl: Multistart optimization methods in...

[zxjroger]

Wow, thanks. I will try this soon.