# Differentiating through optimization problem

**URL:** <https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435>\
**Category:** New to Julia\
**Tags:** question, autodiff, enzyme\
**Created:** [May 18, 2024, 3:47pm UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435 "2024-05-18T15:47:21Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![ij05](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ij05/32/209328_2.png) [@ij05](https://discourse.julialang.org/u/ij05)\
**Post date:** [May 18, 2024, 3:47pm UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/1 "2024-05-18T15:47:21Z")

</div>

Hello all,  
I am very new to Julia, and I intend to migrate from Matlab to Julia. I was very intrigued with automatic differentiation, and worked out a few problems by hand, to understand the concepts behind reverse mode and forward mode AD. But in all these, it seems that you need to have a function in hand (say f(x,y) = x^2+x\*y).

I have the following issue though, I have an equation lc, which is a function of six variables: px, py, pz, phi, theta, psi. Here, the variables phi, px, py and pz are dependent on psi and theta, but I do not have their mathematical expressions. Is it possible to implement reverse AD to find the derivatives of lc, with respect to psi and theta? How to do so in enzyme? I profusely apologize for the lack of MWE.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 18, 2024, 3:57pm UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/2 "2024-05-18T15:57:53Z")

</div>

Welcome!  
What you are looking for is called implicit differentiation: differentiating through a function which is given “implicitly”, by an equation.  
I have implemented this in Julia with the package ImplicitDifferentiation.jl, I encourage you to take a look and see if you manage to use it. If not, post some self-contained code on here with your equation and the solver you want to apply, and I’ll help you!  
Note that my package is not yet compatible with Enzyme.jl, so you will have to perform the differentiation with Zygote.jl or ForwardDiff.jl for now.

---

<div class="post-metadata">

**Author:** ![ij05](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ij05/32/209328_2.png) [@ij05](https://discourse.julialang.org/u/ij05)\
**Post date:** [May 18, 2024, 4:14pm UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/3 "2024-05-18T16:14:27Z")

</div>

Hello, thank you very much for replying!. I shall go through the package ImplicitDifferentiation.jl and let me see how much I can progress. I will get back with my progress.  
Thanks again, for your support!

---

<div class="post-metadata">

**Author:** ![ij05](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ij05/32/209328_2.png) [@ij05](https://discourse.julialang.org/u/ij05)\
**Post date:** [May 19, 2024, 7:30am UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/5 "2024-05-19T07:30:14Z")

</div>

Hello again, I have gone through the package [ImplicitDifferentiation.jl](https://juliadecisionfocusedlearning.github.io/ImplicitDifferentiation.jl/stable/#ImplicitDifferentiation.jl), and I am not quite sure if this is what I require, for my problem. The function that I want to differentiate is:  
`lc=(((-1).*PZ+(-1).*br2.*cos(psi).*sin(th)).^2+((-1).*PX+(-1).*br2.*((-1).*cos(th).*sin(phi)+cos(phi).*sin(psi).*sin(th))).^2+(br2+(-1).*PY+(-1).*br2.*(cos(phi).*cos(th)+sin(phi).*sin(psi).*sin(th))).^2).^(1/2)`

Here, only the numerical values of px,py,pz and phi are known, and are obtained from a blackbox. Their relationship to th and psi are not known. The numerical value of lc, on a grid of th,psi is depicted in Fig.1.

 ![The numerical value of lc, at each point in the 2D grid of coordinates (th,psi)](https://global.discourse-cdn.com/julialang/original/3X/2/9/298f9520c42b37d09199b6e6b83827a2cd8c567c.png)

The values of [th,psi,phi,px,py,pz,lc], for the second row, (\psi=0) from left to right is:

`[-0.0174533 0.0 -4.55313e-17 6.72838e-15 1.33124 152.659 151.182; 0.0 0.0 3.1262e-17 1.641e-14 2.84933e-14 152.665 152.665; 0.0174533 0.0 -6.9987e-17 -1.32915e-15 -1.33323 152.659 154.149] `

I wish to find \frac{\partial lc}{\partial\theta}, at [\theta,\psi]=[0,0]. Numerically, I can use the central difference differentiation. \frac{\partial lc}{\partial\theta}=\frac{lc(\theta=\frac{\pi}{180},\psi=0)-lc(\theta=\frac{-\pi}{180},\psi=0)}{\frac{2\pi}{180}} = (154.149-151.182)/(2\*pi/180) = 84.9983.

Is there any possibility that I can use ImplicitDifferentiation to determine the partial derivatives of lc, with respect to \theta and \psi, at any point in the 2D grid? I would deeply appreciate any insights or pointers in this.  
-Thank you very much!

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 19, 2024, 8:07am UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/6 "2024-05-19T08:07:16Z")

</div>

I had misunderstood your problem. When you said

> [@ij05](#):
>
> Here, the variables phi, px, py and pz are dependent on psi and theta, but I do not have their mathematical expressions.

I thought you meant that the dependence of (\phi, p\_x, p\_y, p\_z) on (\psi, \theta) was defined _by an equation_ \mathrm{lc}(\phi, p\_x, p\_y, p\_z, \psi, \theta) = 0.

But apparently that’s not the case. Instead, you want to differentiate \mathrm{lc}(\phi(\psi, \theta), p\_x(\psi, \theta), p\_y(\psi, \theta), p\_z(\psi, \theta), \psi, \theta) with respect to \theta, but you only have access to the functions \phi and friends through a “black box”, which you call on a grid.  
What does this black box look like? Is it coded in Julia? Can you call it outside of the grid?

---

<div class="post-metadata">

**Author:** ![ij05](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ij05/32/209328_2.png) [@ij05](https://discourse.julialang.org/u/ij05)\
**Post date:** [May 19, 2024, 8:44am UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/7 "2024-05-19T08:44:07Z")

</div>

The \phi and friends are obtained from a least squares optimizer, which I consider as a black box. The variables \theta and \psi goes into the black box that return \phi,~p\_x,~p\_y,~p\_z. Only their numerical values are available, which are to be substituted into the expression lc.

**Edit 1:**

> Can you call it outside of the grid?

I am not sure if I correctly understood the question, but the numerical values of the variables: \phi,~p\_x,~p\_y,~p\_z are available on demand.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 19, 2024, 9:41am UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/8 "2024-05-19T09:41:23Z")

</div>

> [@ij05](#):
>
> a least squares optimizer, which I consider as a black box.

That is interesting, because you can apply implicit differentiation there. Can you give me the formula for the least squares problem?

> [@ij05](#):
>
> the numerical values of the variables: \phi,~p\_x,~p\_y,~p\_zϕ, px, py, pz\phi,~p\_x,~p\_y,~p\_z are available on demand.

In which programming language is the code that produces them? Is it a Julia code?

---

<div class="post-metadata">

**Author:** ![ij05](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ij05/32/209328_2.png) [@ij05](https://discourse.julialang.org/u/ij05)\
**Post date:** [May 19, 2024, 10:32am UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/9 "2024-05-19T10:32:39Z")

</div>

Thank you for your continuous inputs. The vector that I pass into LeastSquaresOptim has 24 very long entries, which I think would be very difficult to be displayed here. This is basically an inverse kinematics problem of a mechanism, where the two input parameters are \theta and \psi, and the remaining 4 parameters \phi,~p\_x,~p\_y,~p\_z are solved by minimizing the least squares of the constraint equations.  
Your suggestion to do the implicit differentiation in the blackbox seem interesting, as the general nature of the constraint equations are of the form:  
\Gamma\_i(\beta\_1(\theta,\psi),\beta\_2(\theta,\psi)\cdots\beta\_{11}(\theta,\psi)\cdots p\_x(\theta,\psi),p\_y(\theta,\psi),p\_z(\theta,\psi),\phi(\theta,\psi))=0 \quad i=1:24  
Upon giving the parameters \theta,~\psi and the initial conditions for the 15 variables in the function, the least-squares optimizer would return all the 15 parameters, out of which I use p\_x(\theta,\psi),p\_y(\theta,\psi),p\_z(\theta,\psi),\phi(\theta,\psi) and \theta,\psi to compute the function lc.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 19, 2024, 1:43pm UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/10 "2024-05-19T13:43:39Z")

</div>

I didn’t know your least squares solver was a Julia package! In that case, and since you only have very few parameters, I would recommend direct automatic differentiation of the solver in forward mode. It won’t be as efficient as implicit differentiation, but it will be 100x easier to code. ForwardDiff.jl is the package that is most likely to work here (but Enzyme.jl might too). If you have a function

```julia
f([theta, psi]) -> [phi, px, py, pz] # calls the least squares solver under the hood

```

can you just try

```julia
using ForwardDiff
ForwardDiff.jacobian(f, [theta, psi])

```

and see what it does?

---

<div class="post-metadata">

**Author:** ![avikpal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/avikpal/32/6550_2.png) [@avikpal](https://discourse.julialang.org/u/avikpal)\
**Post date:** [May 19, 2024, 2:36pm UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/11 "2024-05-19T14:36:13Z")

</div>

If you rewrite the problem using NonlinearLeastSquaresProblem from NonlinearSolve, forwarddiff does have the correct rules for it. Direct AD of nonlinear systems tends to have very little to no benefits.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 19, 2024, 5:16pm UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/12 "2024-05-19T17:16:50Z")

</div>

That is of course an even better solution

---

<div class="post-metadata">

**Author:** ![zdenek\_hurak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zdenek_hurak/32/53118_2.png) [@zdenek\_hurak](https://discourse.julialang.org/u/zdenek_hurak)\
**Post date:** [May 19, 2024, 5:29pm UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/13 "2024-05-19T17:29:17Z")

</div>

> [@ij05](#):
>
> … the concepts behind reverse mode and forward mode AD. But in all these, it seems that you need to have a function in hand (say f(x,y) = x^2+x\*y).

Actually no (as long as by “having a function in hand” you mean having a symbolic prescription for the function). As is now perhaps obvious from the other responses, AD can differentiate a code! Details of software implementation matter, that is why they asked you if your functions were implemented in Julia. But in principle, yes, you can find a derivative of an outcome from some numerical solver with respect to its inputs even if you do not have a formula.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 19, 2024, 5:49pm UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/14 "2024-05-19T17:49:24Z")

</div>

> [@zdenek\_hurak](#):
>
> yes, you can find a derivative of an outcome from some numerical solver with respect to its inputs even if you do not have a formula.

And to sum up, for numerical solvers, there are two main ways to do this:

- the direct way (piggyback autodiff through the solver)
- the implicit way (exploit what we know about the output)

The implicit way, when available, is the clever one, and here I see no reason why it wouldn’t work with @avikpal’s suggestion

---

<div class="post-metadata">

**Author:** ![ij05](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ij05/32/209328_2.png) [@ij05](https://discourse.julialang.org/u/ij05)\
**Post date:** [May 19, 2024, 6:03pm UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/15 "2024-05-19T18:03:51Z")

</div>

> I didn’t know your least squares solver was a Julia package!

Apologies for missing this information a miss in my earlier posts. I have a bit of a difficulty in understanding what you meant by:  
`f([theta, psi]) -> [phi, px, py, pz] # calls the least squares solver under the hood can you just try`  
Did you mean the function `lc` whose partial derivatives are required to be found out? Is it correct if I implement the following?

```julia
lc = (theta,psi) -> [phi,px,py,pz] #calling the least square solver under the hood
using ForwardDiff
ForwardDiff.Jacobian(lc,[theta,psi])

```

@zdenek_hurak Thank you for your valuable insights, as someone who recently stumbled into the concept of automatic differentiation, knowing that I can still find the derivative of what a numerical solver returns, with respect to its inputs is a revelation. I am yet to figure out how to get this implemented.

---

<div class="post-metadata">

**Author:** ![ij05](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ij05/32/209328_2.png) [@ij05](https://discourse.julialang.org/u/ij05)\
**Post date:** [May 19, 2024, 6:44pm UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/16 "2024-05-19T18:44:42Z")

</div>

Thanks for your reply, which is quite encouraging for me!. I however am having a hard time figuring out how to implement this. I think I understand the first part, which is to rewrite the problem from LeastSquaresOptim, to NonLinearLeastSquaresProblem. After this, how can I use forwarddiff to evaluate \frac{\partial lc}{\partial\theta} and \frac{\partial lc}{\partial\theta}? I would be deeply grateful if you ould share a rough syntax. I still cannot understand how to wrap the solver and pass it into forwarddiff, to obtain the derivative/jacobian.

---

<div class="post-metadata">

**Author:** ![avikpal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/avikpal/32/6550_2.png) [@avikpal](https://discourse.julialang.org/u/avikpal)\
**Post date:** [May 19, 2024, 8:56pm UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/17 "2024-05-19T20:56:33Z")

</div>

See section 2.3 [https://arxiv.org/pdf/2403.16341#page=7.12](https://arxiv.org/pdf/2403.16341#page=7.12). Once you rewrite the problem, you can also use LeastSquaresOptim using [LeastSquaresOptim.jl · NonlinearSolve.jl](https://docs.sciml.ai/NonlinearSolve/stable/api/leastsquaresoptim/)

---

<div class="post-metadata">

**Author:** ![ij05](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ij05/32/209328_2.png) [@ij05](https://discourse.julialang.org/u/ij05)\
**Post date:** [May 20, 2024, 6:12am UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/18 "2024-05-20T06:12:03Z")

</div>

@avikpal , Thank you very much for sharing!. I can work with this. I shall report back later with my experiences.  
@gdalle, I am curious to try the implicit way as well to compare the efficiency. However, I am also a bit concerned as you say it is 100x easier to go with AD. If possible, could you share a rough skeleton on how to implement it?

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [May 20, 2024, 1:29pm UTC](https://discourse.julialang.org/t/differentiating-through-optimization-problem/114435/19 "2024-05-20T13:29:34Z")

</div>

The implicit/clever way is exactly what @avikpal suggests: taking advantage of our knowledge that the output is a solution to a nonlinear system of equations. My package ImplicitDifferentiation.jl would allow you to do it manually because it is more generic, but NonlinearSolve.jl does it automatically for the special case you need here. It contains custom differentiation rules that are essentially equivalent to the implicit function theorem.