# Need Diffractor.jl for State-of-the-Art Deep Learning Model

**URL:** <https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423>\
**Category:** Machine Learning\
**Tags:** differentiation, flux, zygote\
**Created:** [April 16, 2021, 4:37pm UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423 "2021-04-16T16:37:35Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Freddy\_Chua](https://avatars.discourse-cdn.com/v4/letter/f/bc8723/32.png) [@Freddy\_Chua](https://discourse.julialang.org/u/Freddy_Chua)\
**Post date:** [April 16, 2021, 4:37pm UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423/1 "2021-04-16T16:37:35Z")

</div>

Hi, I hope to beta test for your Diffractor.jl next generation AD.

I am a co-author of DeepCPCFG [[2103.05908] DeepCPCFG: Deep Learning and Context Free Grammars for End-to-End Information Extraction](https://arxiv.org/abs/2103.05908).  
This research is also a guest lecture at MIT Intro to Deep Learning [MIT 6.S191: Deep CPCFG for Information Extraction - YouTube](https://www.youtube.com/watch?v=WkUYsVC3hKI&t=1701s)

I have experienced some memory leak issues while using Zygote.jl to implement the above ML model that combines dynamic programming and deep neural network. I have provided the MWE here [https://github.com/deepcpcfg/zygote\_memleak](https://github.com/deepcpcfg/zygote_memleak)

and the issue is here [https://github.com/FluxML/Zygote.jl/issues/930](https://github.com/FluxML/Zygote.jl/issues/930)

To overcome this issue, I have to resort to using Tracker.jl

I am requesting to beta test diffractor.jl

@Keno @dhairyagandhi96 @darsnack

---

<div class="post-metadata">

**Author:** ![DoktorMike](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/doktormike/32/2736_2.png) [@DoktorMike](https://discourse.julialang.org/u/DoktorMike)\
**Post date:** [April 16, 2021, 5:40pm UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423/2 "2021-04-16T17:40:49Z")

</div>

I’ve whispers of this diffractor.jl many places now but never seen a dev/wip repo for it. I’m anxious to try it too should it ever come into excistance. 😊

---

<div class="post-metadata">

**Author:** ![sethaxen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sethaxen/32/35604_2.png) [@sethaxen](https://discourse.julialang.org/u/sethaxen)\
**Post date:** [July 16, 2021, 10:13am UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423/3 "2021-07-16T10:13:40Z")

</div>

Word has it that it will be released at JuliaCon.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [July 16, 2021, 11:23am UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423/4 "2021-07-16T11:23:42Z")

</div>

I am looking forward so much.

---

<div class="post-metadata">

**Author:** ![DoktorMike](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/doktormike/32/2736_2.png) [@DoktorMike](https://discourse.julialang.org/u/DoktorMike)\
**Post date:** [July 16, 2021, 6:38pm UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423/5 "2021-07-16T18:38:24Z")

</div>

Can’t wait. 💪🏻👍🏻✅😊

---

<div class="post-metadata">

**Author:** ![rkube](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rkube/32/211198_2.png) [@rkube](https://discourse.julialang.org/u/rkube)\
**Post date:** [July 26, 2021, 2:19pm UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423/6 "2021-07-26T14:19:25Z")

</div>

[https://github.com/Keno/Diffractor.jl](https://github.com/Keno/Diffractor.jl)

---

<div class="post-metadata">

**Author:** ![Freddy\_Chua](https://avatars.discourse-cdn.com/v4/letter/f/bc8723/32.png) [@Freddy\_Chua](https://discourse.julialang.org/u/Freddy_Chua)\
**Post date:** [July 27, 2021, 6:47am UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423/7 "2021-07-27T06:47:09Z")

</div>

We are witnessing a milestone in the history of Computer Science.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [July 27, 2021, 9:32am UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423/8 "2021-07-27T09:32:55Z")

</div>

> We are witnessing a milestone in the history of Computer Science.

😂 cool your jets\* ace.  
Diffractor is pretty exciting, but at the end of the day it is not world changing.

It’s probably going to have many bugs – but different bugs to Zygote.

Still I also am pretty hype.  
We are indeed going to be able to do a lot of cool things faster, especially when it comes to derivatives of functions of derivatives.

(\*Pun not intented, but kinda good: [Jets](https://en.wikipedia.org/wiki/Jet_(mathematics)) are used for higher order AD, which Diffractor does)

---

<div class="post-metadata">

**Author:** ![1115](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/1115/32/4465_2.png) [@1115](https://discourse.julialang.org/u/1115)\
**Post date:** [July 27, 2021, 10:36am UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423/9 "2021-07-27T10:36:22Z")

</div>

I can see some tests about higher order AD in Diffractor, it is a good proof of its effort in optimzing compiling. Just wondering what is the typical using case of higher order reverse mode AD?

For 2nd order, we just use Forward over backward.  
For 3 order or higher, I can not see much advantage of using Reverse mode AD, because the input dimension can not be large.

Is it just for checking the compiler performance?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 27, 2021, 10:40am UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423/10 "2021-07-27T10:40:35Z")

</div>

> [@1115](#):
>
> I can see some tests about higher order AD in Diffractor, it is a good proof of its effort in optimzing compiling. Just wondering what is the typical using case of higher order reverse mode AD?

There is a use case for 2 reverse passes (but not more, the rest forward) in some applications of physics-informed neural networks, and so you can see PINNs as the test case. Fun fact, Diffractor.jl was grant funded for these weird SciML applications (same ARPA-E that made ModelingToolkit.jl, Symbolics.jl, and JuliaSim), but of course fast higher order AD is something anyone could benefit from.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [July 27, 2021, 10:43am UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423/11 "2021-07-27T10:43:43Z")

</div>

I agree for vanillia higher order AD, there are better ways than nesting reverse.  
E.g. Forward over forward, Forward over reverse (especially for hessians), and taylor mode.  
I complained to Keno about this several times, but i have now been convinced.

The case Diffactor is useful for is not directly nested AD but when there is a function in between.  
i.e AD → some function of gradients → AD

For example imagine that I have some back box `bb(x)` and I am trying to train a Neural net to imitate it `nn(x; θ)` (where theta is my neural network parameters).  
And I not only want to get the output to match, I also want the derivitive to match.  
That will mean each training example is much more informative – which is good if `bb` is expensive to run (which it quite probably is, since that is one reason to train a nn to immitate it)

So I write my loss function as as the sum of the squared difference both of `bb` and `nn` at `x`, and of their derivatives `bb'` and `nn'` at `x`

```julia
loss(x, θ) = (bb(x) - nn(x; θ))^2 + (bb'(x) - nn'(x; θ))^2

```

Now I want to compute the dervative of `loss` with regards to `θ` so I can train my neural net.  
Done naively the result is that one would call reverse mode AD on code generated by reverse mode AD.

Maybe there is a smart way to reformulate it to avoid calling reverse on code generated by reverse.  
If so, I would like to hear about it. (especially if it is generalizable to other problems that are not quite in this form)

---

<div class="post-metadata">

**Author:** ![1115](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/1115/32/4465_2.png) [@1115](https://discourse.julialang.org/u/1115)\
**Post date:** [July 27, 2021, 10:55am UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423/12 "2021-07-27T10:55:17Z")

</div>

Thanks for your example, it looks like a decent using case for nested reverse mode AD.  
This remembers me in physics, we often switch the description between energy, forces and acceleration. When including acceleration in the loss, it is a 3rd order reverse mode AD w.r.t. the energy model. Makes sense.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 27, 2021, 11:03am UTC](https://discourse.julialang.org/t/need-diffractor-jl-for-state-of-the-art-deep-learning-model/59423/13 "2021-07-27T11:03:15Z")

</div>

> [@oxinabox](#):
>
> So I write my loss function as as the sum of the squared difference both of `bb` and `nn` at `x` , and of their derivatives `bb'` and `nn'` at `x`

If NN is N → 1 sized, then you want to do reverse mode for the x values per x in the loss function, but the n train doing reverse mode once for the theta. That’s then a reason for reverse-over-reverse. Then for the second derivative, N-\>1 becomes N-\>N after the first, in that case you then want to do forward-over-reverse-over-reverse. This is exactly the PINN reasons BTW.