# Comparison of Automatic Differentiation (AD)?

**URL:** <https://discourse.julialang.org/t/comparison-of-automatic-differentiation-ad/91544>\
**Category:** Optimization (Mathematical)\
**Created:** [December 12, 2022, 8:20am UTC](https://discourse.julialang.org/t/comparison-of-automatic-differentiation-ad/91544 "2022-12-12T08:20:09Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![bdas123](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bdas123/32/32142_2.png) [@bdas123](https://discourse.julialang.org/u/bdas123)\
**Post date:** [December 12, 2022, 8:20am UTC](https://discourse.julialang.org/t/comparison-of-automatic-differentiation-ad/91544/1 "2022-12-12T08:20:09Z")

</div>

Hello! Has there been a recent study (within the past year) that compares the speed/performance of difference Automatic Differentiation packages (Zygote, ReverseDiff, etc.)?

---

<div class="post-metadata">

**Author:** ![nrontsis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nrontsis/32/38535_2.png) [@nrontsis](https://discourse.julialang.org/u/nrontsis)\
**Post date:** [December 12, 2022, 11:11am UTC](https://discourse.julialang.org/t/comparison-of-automatic-differentiation-ad/91544/2 "2022-12-12T11:11:01Z")

</div>

Related posts:

> **[Engineering Trade-Offs in Automatic Differentiation: from TensorFlow and...](http://www.stochasticlifestyle.com/engineering-trade-offs-in-automatic-differentiation-from-tensorflow-and-pytorch-to-jax-and-julia/)**
>
> To understand the differences between automatic differentiation libraries, let’s talk about the engineering trade-offs that were made. I would personally say that none of these libraries are “better” than another, they simply all make engineering...

> [@Open discussion on the state of differentiable physics in Julia](https://discourse.julialang.org/t/open-discussion-on-the-state-of-differentiable-physics-in-julia/72900/2):
>
> I knew you would ask because you’re hitting THE pain point I know about right now. From the [Universal Differential Equations paper supplement](https://arxiv.org/abs/2001.04385) it describes the 4 vjp choices: Zygote.jl source-to-source transformation based vjps. Note that only non-mutating differential equation function definitions are supported in this mode. This mode is the most efficient in the presence of neural networks. Enzyme.jl source-to-source transformation basd vjps. This is the fastest vjp choice in the presence of …

And [this paper](https://arxiv.org/abs/1812.01892).

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [December 12, 2022, 6:45pm UTC](https://discourse.julialang.org/t/comparison-of-automatic-differentiation-ad/91544/3 "2022-12-12T18:45:33Z")

</div>

> Has there been a recent study (within the past year) that compares the speed/performance of difference Automatic Differentiation packages (Zygote, ReverseDiff, etc.)?

The answer is going to be “it depends.” Different packages/algorithms have their strengths and weaknesses. What is your use-case?

---

<div class="post-metadata">

**Author:** ![bdas123](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bdas123/32/32142_2.png) [@bdas123](https://discourse.julialang.org/u/bdas123)\
**Post date:** [December 14, 2022, 5:06am UTC](https://discourse.julialang.org/t/comparison-of-automatic-differentiation-ad/91544/4 "2022-12-14T05:06:45Z")

</div>

I’m trying to fit some parameters within a model to data.

Right now, it’s in the thousands, but I estimate that this model can go in the tens of thousands or hundreds of thousands.

---

<div class="post-metadata">

**Author:** ![JesperMartinsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jespermartinsson/32/34098_2.png) [@JesperMartinsson](https://discourse.julialang.org/u/JesperMartinsson)\
**Post date:** [December 14, 2022, 7:03am UTC](https://discourse.julialang.org/t/comparison-of-automatic-differentiation-ad/91544/5 "2022-12-14T07:03:20Z")

</div>

With that many parameters you might perhaps resort to gradient free methods. Perhaps a Gibbs sampling strategy would be efficient where the sampler can evaluate one parameter at a time while keeping the other constant. You might be able to factor out your likelihood in many small different parts (each depending on just a few parameters for your Gibbs evaluation), that will speed evaluation of your posterior up a lot (e.g. by not evaluating parts you do not need to in Gibbs).

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [December 14, 2022, 7:54am UTC](https://discourse.julialang.org/t/comparison-of-automatic-differentiation-ad/91544/6 "2022-12-14T07:54:32Z")

</div>

> I’m trying to fit some parameters within a model to data.

How? What is the model?

> [@JesperMartinsson](#):
>
> With that many parameters you might perhaps resort to gradient free methods.

Depending on the properties of the model you’re trying to fit, gradient based methods work fine for 10^4 or 10^5 parameters. See, e.g., JuMP on a nonlinear programming problem from optimal power flow with 10^5+ variables and constraints: [https://youtu.be/tvBNQcuU-hY?t=997](https://youtu.be/tvBNQcuU-hY?t=997).

---

<div class="post-metadata">

**Author:** ![JesperMartinsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jespermartinsson/32/34098_2.png) [@JesperMartinsson](https://discourse.julialang.org/u/JesperMartinsson)\
**Post date:** [December 14, 2022, 11:05pm UTC](https://discourse.julialang.org/t/comparison-of-automatic-differentiation-ad/91544/7 "2022-12-14T23:05:22Z")

</div>

We need to know a bit more details about the OP’s model and objective function to find suitable strategies. E.g. is factorization of the likelihood into smaller parts impossible, then Gibbs (or slice) sampling do not provide gains (in evaluating the joint posterior) over other more sophisticated sampling strategies (gradient or gradient free). Perhaps the optimization/estimation problem is difficult to express in terms of likelihood (and prior)?