# Does automatic differentiation work for function with discrete argument?

**URL:** <https://discourse.julialang.org/t/does-automatic-differentiation-work-for-function-with-discrete-argument/35947>\
**Category:** Optimization (Mathematical)\
**Created:** [March 13, 2020, 9:03pm UTC](https://discourse.julialang.org/t/does-automatic-differentiation-work-for-function-with-discrete-argument/35947 "2020-03-13T21:03:01Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![zxjroger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zxjroger/32/11826_2.png) [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Post date:** [March 13, 2020, 9:03pm UTC](https://discourse.julialang.org/t/does-automatic-differentiation-work-for-function-with-discrete-argument/35947/1 "2020-03-13T21:03:01Z")

</div>

I am trying to find the minimizer \theta of the following nonlinear and non-differentiable function: g(H(X | \theta),X). Here, X is known and imported from dataset; \theta=(\theta\_1,\theta\_2,\theta\_3) are the parameters I want to solve for; H is discrete and therefore makes g(\cdot) non-differentiable with respect to \theta. There is only one simple constraint such that the first parameter \theta\_1 is positive.

Since the argument H is discrete, I wonder if I can still use automatic differentiation in Ipopt for this optimization problem? Thank you.

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [March 13, 2020, 9:39pm UTC](https://discourse.julialang.org/t/does-automatic-differentiation-work-for-function-with-discrete-argument/35947/2 "2020-03-13T21:39:29Z")

</div>

Well, yes and no. Automatic differentiation can handle discrete functions, but all it will do is (correctly) tell you that the derivative is zero:

```julia
julia> using ForwardDiff
[Info: Precompiling ForwardDiff [f6369f11-7733-5829-9624-2563aa707210]
funct     
julia> function f(x)
         if x > 0
           return 1
         else
           return 2
         end
       end
f (generic function with 1 method)

julia> ForwardDiff.derivative(f, 1.0)
0

```

So using autodiff with an optimization method that expects a gradient is unlikely to work well. Perhaps one of the non-gradient-based algorithms from NLOpt would be more appropriate here? [NLopt Algorithms - NLopt Documentation](https://nlopt.readthedocs.io/en/latest/NLopt_Algorithms/)

---

<div class="post-metadata">

**Author:** ![zxjroger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zxjroger/32/11826_2.png) [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Post date:** [March 13, 2020, 10:27pm UTC](https://discourse.julialang.org/t/does-automatic-differentiation-work-for-function-with-discrete-argument/35947/3 "2020-03-13T22:27:07Z")

</div>

I think my function is different from this. The function g(\cdot) is kind of smooth, although argument H is discrete. In your example, function f(\cdot) is discrete but argument x is smooth.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 14, 2020, 6:27am UTC](https://discourse.julialang.org/t/does-automatic-differentiation-work-for-function-with-discrete-argument/35947/4 "2020-03-14T06:27:24Z")

</div>

> [@zxjroger](#):
>
> The function g(⋅)g(\cdot) is kind of smooth, although argument HH is discrete.

But the function you are dealing with is f(\theta) = g(H(X \mid \theta), X) which in general will not be continuous(ly differentiable) if H isn’t, even if g is.

Depending on how bad things are, stochastic (eg simulated annealing) or robust deterministic derivative-free optimization methods are used in these cases. See

> [@Latest recommendations for global optimization](https://discourse.julialang.org/t/latest-recommendations-for-global-optimization/17718):
>
> What are people’s recommendations for a global optimizer? I’m trying to do simulated method of moments (choosing a set of parameters for an economic model to minimise the distance between model predictions and data) and have so far been using blackboxoptim. I find it somewhat unreliable: running exactly the same code on the same data I find sometimes the optimizer gives me a parameter choice that provides a close fit to the data, but sometimes it just seems to get stuck without converging). Any …

> [@Global optimization: Simulated Method of Moments](https://discourse.julialang.org/t/global-optimization-simulated-method-of-moments/23844):
>
> I work with non-linear models that need to be calibrated to match data moments. The setup is simple. I have a function that takes a set of parameters as input (for example, a vector of floats), solves the model, and returns a measure of the distance between the model-generated moments and the data moments. Typically there are more moments than parameters. Hence, I use some simple weighting function to summarize the differences between model and data moments into one float. The objective is to m…

---

<div class="post-metadata">

**Author:** ![zxjroger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zxjroger/32/11826_2.png) [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Post date:** [March 14, 2020, 7:36pm UTC](https://discourse.julialang.org/t/does-automatic-differentiation-work-for-function-with-discrete-argument/35947/5 "2020-03-14T19:36:21Z")

</div>

Thanks for your comments. I have read these two posts previously. My problem is also simulated method of moments. I was testing blackbox and showed these two posts to my friend. But he still wants me to test Ipopt + user-defined numerical derivatives in Julia.

BTW, do you happen to know this new global optimizer named TikTak, which is developed by [Fatih Guvenen](https://fguvenendotcom.files.wordpress.com/2019/09/agk2019-september-nber-submit.pdf)?

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [March 14, 2020, 9:48pm UTC](https://discourse.julialang.org/t/does-automatic-differentiation-work-for-function-with-discrete-argument/35947/6 "2020-03-14T21:48:57Z")

</div>

See [Nonlinear Modeling · JuMP](https://www.juliaopt.org/JuMP.jl/stable/nlp/#User-defined-Functions-1). You can pass a function that evaluates the gradients.

---

<div class="post-metadata">

**Author:** ![zxjroger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zxjroger/32/11826_2.png) [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Post date:** [March 14, 2020, 10:20pm UTC](https://discourse.julialang.org/t/does-automatic-differentiation-work-for-function-with-discrete-argument/35947/7 "2020-03-14T22:20:30Z")

</div>

I do not think it will apply to my case. Basically, it only supports univariate function. I am using MathOptInterface instead and call Ipopt with self-defined derivatives.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 15, 2020, 8:13am UTC](https://discourse.julialang.org/t/does-automatic-differentiation-work-for-function-with-discrete-argument/35947/8 "2020-03-15T08:13:30Z")

</div>

> [@zxjroger](#):
>
> this new global optimizer named TikTak

I implemented a Julia version of it in

> **[GitHub - tpapp/MultistartOptimization.jl: Multistart optimization methods in...](https://github.com/tpapp/MultistartOptimization.jl)**
>
> Multistart optimization methods in Julia. Contribute to tpapp/MultistartOptimization.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![zxjroger](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zxjroger/32/11826_2.png) [@zxjroger](https://discourse.julialang.org/u/zxjroger)\
**Post date:** [March 16, 2020, 1:10am UTC](https://discourse.julialang.org/t/does-automatic-differentiation-work-for-function-with-discrete-argument/35947/9 "2020-03-16T01:10:13Z")

</div>

Wow, thanks. I will try this soon.