# Anybody working on Automatic Differentiation "the Conal Elliot way"?

**URL:** <https://discourse.julialang.org/t/anybody-working-on-automatic-differentiation-the-conal-elliot-way/19887>\
**Category:** Machine Learning\
**Tags:** question\
**Created:** [January 21, 2019, 5:08pm UTC](https://discourse.julialang.org/t/anybody-working-on-automatic-differentiation-the-conal-elliot-way/19887 "2019-01-21T17:08:35Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jbrea](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jbrea/32/3879_2.png) [@jbrea](https://discourse.julialang.org/u/jbrea)\
**Post date:** [January 21, 2019, 5:08pm UTC](https://discourse.julialang.org/t/anybody-working-on-automatic-differentiation-the-conal-elliot-way/19887/1 "2019-01-21T17:08:36Z")

</div>

Just learned about [The simple essence of automatic differentiation](http://conal.net/papers/essence-of-ad/).  
Did somebody already try to implement these ideas in Julia?  
Any thoughts about pros and cons of this approach relative to e.g. [Zygote.jl](https://github.com/FluxML/Zygote.jl)?

---

<div class="post-metadata">

**Author:** ![MikeInnes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikeinnes/32/3656_2.png) [@MikeInnes](https://discourse.julialang.org/u/MikeInnes)\
**Post date:** [January 21, 2019, 5:35pm UTC](https://discourse.julialang.org/t/anybody-working-on-automatic-differentiation-the-conal-elliot-way/19887/2 "2019-01-21T17:35:32Z")

</div>

I haven’t looked in great detail but from what I remember, I don’t think there’s really “an approach” here, so much as a translation of known ideas in AD (e.g. reversal of a Wengert list) to the language of category theory (e.g. taking the dual of a category). Maybe for some people that makes it seem simpler, but I’m not in that boat myself; I’d be interested in seeing something more implementation-focused though.

---

<div class="post-metadata">

**Author:** ![jbrea](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jbrea/32/3879_2.png) [@jbrea](https://discourse.julialang.org/u/jbrea)\
**Post date:** [January 21, 2019, 7:31pm UTC](https://discourse.julialang.org/t/anybody-working-on-automatic-differentiation-the-conal-elliot-way/19887/3 "2019-01-21T19:31:47Z")

</div>

I got the impression that [tapes (Wengert lists) are not needed in Elliot’s approach](https://memento.epfl.ch/event/ic-colloquium-automatic-differentiation-for-dumm-2/); but I am not an expert.

---

<div class="post-metadata">

**Author:** ![Keno](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keno/32/285_2.png) [@Keno](https://discourse.julialang.org/u/Keno)\
**Post date:** [January 21, 2019, 8:43pm UTC](https://discourse.julialang.org/t/anybody-working-on-automatic-differentiation-the-conal-elliot-way/19887/4 "2019-01-21T20:43:56Z")

</div>

You always need to store intermediate values somewhere. There’s lots of fun ways to hide them (e.g. in closure structures which get chained), but you fundamentally can’t get around that (well, you can re-compute, but at that point you’re not really doing reverse mode anymore). Something else people sometimes mean when they say tape-free is that the operations themselves don’t get recorded on the tape, because the operation is implicit (in the code generated by the AD framework), but that’s not really specific to the formulation. FWIW, I do like the push for more formalism in AD, particularly as it relates to connecting to normal notions in differential geometry. However, I also think that it generally has little impact on implementation.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [January 22, 2019, 12:25pm UTC](https://discourse.julialang.org/t/anybody-working-on-automatic-differentiation-the-conal-elliot-way/19887/5 "2019-01-22T12:25:29Z")

</div>

> [@Keno](#):
>
> e.g. in closure structures which get chained

I have been wondering about whether doing this in Julia would be more inference-friendly. But perhaps the chains get too long after a while.

---

<div class="post-metadata">

**Author:** ![Keno](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keno/32/285_2.png) [@Keno](https://discourse.julialang.org/u/Keno)\
**Post date:** [January 23, 2019, 12:11am UTC](https://discourse.julialang.org/t/anybody-working-on-automatic-differentiation-the-conal-elliot-way/19887/6 "2019-01-23T00:11:28Z")

</div>

To some extent yes, which is why Zygote gets better type information (e.g. all the TPU work needs that).