# Taking Complex Autodiff Seriously in ChainRules

**URL:** <https://discourse.julialang.org/t/taking-complex-autodiff-seriously-in-chainrules/39317>\
**Category:** Specific Domains\
**Tags:** numerics, chainrulescore, complex-numbers\
**Created:** [May 12, 2020, 1:52am UTC](https://discourse.julialang.org/t/taking-complex-autodiff-seriously-in-chainrules/39317 "2020-05-12T01:52:35Z")\
**Posts on this page:** 6\
**Page:** 4

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [June 27, 2020, 9:22pm UTC](https://discourse.julialang.org/t/taking-complex-autodiff-seriously-in-chainrules/39317/62 "2020-06-27T21:22:54Z")

</div>

Okay, so after some very productive and stimulating conversation, this recent push has resulting in [ChainRulesCore.jl/pull/167](https://github.com/JuliaDiff/ChainRulesCore.jl/pull/167) (thanks @ettersi!) and [ChainRules.jl/pull/196](https://github.com/JuliaDiff/ChainRules.jl/pull/196) which I must say feels like we’ve come up with a really great set of shared definitions and conventions.

The result of @ettersi’s now merged PR can be viewed in all of it’s LaTeX glory here:  
[https://www.juliadiff.org/ChainRulesCore.jl/dev/complex.html](https://www.juliadiff.org/ChainRulesCore.jl/dev/complex.html)

Perhaps next up, we can lift some ideas from here: [Complex Differentiation · Zygote](https://fluxml.ai/Zygote.jl/dev/complex/) such as the example showing how to get the Wirtinger derivatives, \partial \over \partial z and \partial \over \partial \bar{z} from the Jacobian, i.e.

```julia
using ChainRules
function wirtinger_rrule(f, z)
    _, pullback = rrule(f, z)
    du, dv = pullback(1)[2], pullback(im)[2]
    (du' + im*dv')/2, (du + im*dv)/2
end

wirtinger_rrule(abs2, 1 + im)

#+RESULTS:
: (1.0 - 1.0im, 1.0 + 1.0im)

```

```julia
function wirtinger_frule(f, z)
    du_dx, dv_dx = reim(frule((Zero(), 1),f,z)[2])
    du_dy, dv_dy = reim(frule((Zero(),im),f,z)[2])

    (du_dx + dv_dy + im*(-du_dy + dv_dx))/2, (du_dx - dv_dy + im*(du_dy + dv_dx))/2
end

wirtinger_frule(sin, 1 + im)

#+RESULTS:
: (0.8337300251311491 - 0.9888977057628651im, 0.0 + 0.0im)

```

---

<div class="post-metadata">

**Author:** ![simeonschaub](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simeonschaub/32/216566_2.png) [@simeonschaub](https://discourse.julialang.org/u/simeonschaub)\
**Post date:** [June 27, 2020, 11:30pm UTC](https://discourse.julialang.org/t/taking-complex-autodiff-seriously-in-chainrules/39317/63 "2020-06-27T23:30:48Z")

</div>

Since you point out the Zygote docs, I have an open PR, where I tried to clarify the relationship with Wirtinger derivatives a bit more:

[https://github.com/FluxML/Zygote.jl/pull/328/files](https://github.com/FluxML/Zygote.jl/pull/328/files)

Should be pretty much finished, just got lost in my backlog a bit.

---

<div class="post-metadata">

**Author:** ![ettersi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ettersi/32/6829_2.png) [@ettersi](https://discourse.julialang.org/u/ettersi)\
**Post date:** [July 1, 2020, 2:31am UTC](https://discourse.julialang.org/t/taking-complex-autodiff-seriously-in-chainrules/39317/64 "2020-07-01T02:31:43Z")

</div>

> [@Mason](#):
>
> Perhaps next up, we can lift some ideas from here: [https://fluxml.ai/Zygote.jl/dev/complex/](https://fluxml.ai/Zygote.jl/dev/complex/) such as the example showing how to get the Wirtinger derivatives from the Jacobian,

I am not sure what you mean here. AFAICT, the “only” remaining obstacle to introducing Wirtinger derivatives is to implement the chaining of chain rules as done e.g. in Zygote. Once that is done, Wirtinger derivatives can be provided using either of the functions you mentioned.

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [July 1, 2020, 3:13am UTC](https://discourse.julialang.org/t/taking-complex-autodiff-seriously-in-chainrules/39317/65 "2020-07-01T03:13:30Z")

</div>

Oh I just meant adding that to our documentation. The functions I showed above work today. I’ll try to open a PR soon

---

<div class="post-metadata">

**Author:** ![Arrigo\_Benedetti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arrigo_benedetti/32/25545_2.png) [@Arrigo\_Benedetti](https://discourse.julialang.org/u/Arrigo_Benedetti)\
**Post date:** [July 9, 2020, 7:16pm UTC](https://discourse.julialang.org/t/taking-complex-autodiff-seriously-in-chainrules/39317/66 "2020-07-09T19:16:23Z")

</div>

I am not sure if this paper has been mentioned in this thread already, anyway it is very relevant to this discussion. The authors have also implemented their ideas in Julia code:

[Guo, Chu, and Dario Poletti. “A scheme for automatic differentiation of complex loss functions.”](https://arxiv.org/abs/2003.04295)

> **[A scheme for automatic differentiation of complex loss functions](https://arxiv.org/abs/2003.04295)**
>
> For a real function, automatic differentiation is such a standard algorithm used to efficiently compute its gradient, that it is integrated in various neural network frameworks. However, despite the recent advances in using complex functions in...

---

<div class="post-metadata">

**Author:** ![ettersi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ettersi/32/6829_2.png) [@ettersi](https://discourse.julialang.org/u/ettersi)\
**Post date:** [July 10, 2020, 1:45pm UTC](https://discourse.julialang.org/t/taking-complex-autodiff-seriously-in-chainrules/39317/67 "2020-07-10T13:45:07Z")

</div>

Highly relevant indeed! Thanks for pointing this out.

Their proposal is the same as what we came up with here, just approached from a different angle. That gives further validation that this convention is most likely the right thing to do.

[Previous page](https://discourse.julialang.org/t/taking-complex-autodiff-seriously-in-chainrules/39317.md?page=3)