# Does Zygote differentiate symbolically?

**URL:** <https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528>\
**Category:** Machine Learning\
**Tags:** question, differentiation, zygote\
**Created:** [March 23, 2023, 6:40pm UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528 "2023-03-23T18:40:05Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Euhan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/euhan/32/36548_2.png) [@Euhan](https://discourse.julialang.org/u/Euhan)\
**Post date:** [March 23, 2023, 6:40pm UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/1 "2023-03-23T18:40:06Z")

</div>

Sorry for posting this question in maybe-not-exactly-right category! This was my guess at group with biggest overlap with my question.

My question is as simple as it says in the title. Does Zygote differentiate symbolically?

I would also like to know:

- If it does, can the results be seen in some way?
- Are there other packages that differentiate symbolically + previous question?
- How (and where) does Zygote overload one single quote?

---

<div class="post-metadata">

**Author:** ![contradict](https://avatars.discourse-cdn.com/v4/letter/c/ac91a4/32.png) [@contradict](https://discourse.julialang.org/u/contradict)\
**Post date:** [March 23, 2023, 8:16pm UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/2 "2023-03-23T20:16:06Z")

</div>

> My question is as simple as it says in the title. Does Zygote differentiate symbolically?

No, [Automatic Differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation) is distinct from symbolic differentiation. Zygote can perform forward-mode or reverse-mode automatic differentiation by source rewriting.

> If it does, can the results be seen in some way?

Sort of, although it is not very helpful. There is an [example](https://fluxml.ai/Zygote.jl/stable/internals/#Entry-Points-1) in the documentation.

> Are there other packages that differentiate symbolically + previous question?

[Symbolics.jl](https://symbolics.juliasymbolics.org/dev/manual/derivatives/#Derivatives-and-Differentials) can differentiate symbolic expressions.

> How (and where) does Zygote overload one single quote?

I didn’t know about this! Can you point out an example of the syntax you mean?

---

<div class="post-metadata">

**Author:** ![Euhan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/euhan/32/36548_2.png) [@Euhan](https://discourse.julialang.org/u/Euhan)\
**Post date:** [March 24, 2023, 10:46am UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/3 "2023-03-24T10:46:14Z")

</div>

> [@contradict](#):
>
> I didn’t know about this! Can you point out an example of the syntax you mean?

`f'(x)` denotes the derivative of `f(x)`.

E.g. `sin'(0)` returns `1`.

> [@contradict](#):
>
> No, [Automatic Differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation) is distinct from symbolic differentiation. Zygote can perform forward-mode or reverse-mode automatic differentiation by source rewriting.

In my mind, I imagined that differentiation could be done symbolically or numerically, and that automatic differentiation obviated the need to do it oneself by doing one of them. What does Zygote do exactly (not exactly, a broad outline will do fine)?

---

<div class="post-metadata">

**Author:** ![johnmyleswhite](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnmyleswhite/32/31_2.png) [@johnmyleswhite](https://discourse.julialang.org/u/johnmyleswhite)\
**Post date:** [March 24, 2023, 11:28am UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/4 "2023-03-24T11:28:52Z")

</div>

> [@Euhan](#):
>
> In my mind, I imagined that differentiation could be done symbolically or numerically, and that automatic differentiation obviated the need to do it oneself by doing one of them. What does Zygote do exactly (not exactly, a broad outline will do fine)?

It sounds like you’re not familiar with the concept of AD. It’s worth starting with the simplified problem of using dual numbers: [GitHub - JuliaDiff/DualNumbers.jl: Julia package for representing dual numbers and for performing dual algebra](https://github.com/JuliaDiff/DualNumbers.jl)

Read that code and then read the Wikipedia page on AD: [Automatic differentiation - Wikipedia](https://en.wikipedia.org/wiki/Automatic_differentiation)

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [March 24, 2023, 12:24pm UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/5 "2023-03-24T12:24:03Z")

</div>

Promoting our lectures for students, you read about AD here  
[https://juliateachingctu.github.io/Scientific-Programming-in-Julia/dev/lecture\_08/lecture/](https://juliateachingctu.github.io/Scientific-Programming-in-Julia/dev/lecture_08/lecture/)  
it is meant to be very introductory

---

<div class="post-metadata">

**Author:** ![contradict](https://avatars.discourse-cdn.com/v4/letter/c/ac91a4/32.png) [@contradict](https://discourse.julialang.org/u/contradict)\
**Post date:** [March 24, 2023, 3:13pm UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/6 "2023-03-24T15:13:56Z")

</div>

Oh, that is [`Base.adjoint`](https://docs.julialang.org/en/v1/stdlib/LinearAlgebra/#Base.adjoint), which can also be spelled `'` . The easy way to find the new method is to use `@edit sin'(0)` in a REPL, that will bring up [this](https://github.com/FluxML/Zygote.jl/blob/master/src/compiler/interface.jl#L102) line in your editor.

---

<div class="post-metadata">

**Author:** ![zdenek\_hurak](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zdenek_hurak/32/53118_2.png) [@zdenek\_hurak](https://discourse.julialang.org/u/zdenek_hurak)\
**Post date:** [March 24, 2023, 4:49pm UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/7 "2023-03-24T16:49:28Z")

</div>

> [@Euhan](#):
>
> In my mind, I imagined that differentiation could be done symbolically or numerically, and that automatic differentiation obviated the need to do it oneself by doing one of them.

This is why I prefer the other interpretation of the AD abbreviation – **Algorithmic Differentiation**. I think it better reflects the essence than **Automatic differentiation**. Anyway, AD is distinct both from symbolic and numerical differentiation [Simple numerical differentiation - #5 by zdenek\_hurak](https://discourse.julialang.org/t/simple-numerical-differentiation/52254/5).

---

<div class="post-metadata">

**Author:** ![Euhan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/euhan/32/36548_2.png) [@Euhan](https://discourse.julialang.org/u/Euhan)\
**Post date:** [March 26, 2023, 1:32pm UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/8 "2023-03-26T13:32:57Z")

</div>

OK, I’m going to summarise what I have gleaned from links in the answers (and to some extent the answers themselves). This is so that others can identify any misunderstandings I might be labouring under.

Let me start with refining my question a bit (this is so that we may not have diverging ideas about what is meant in this context by a certain concept or term). I am talking about Julia at my university department (at some point in the near future). I (and I think my audience) want to know whether Zygote uses finite differences to calculate derivatives.

As far as I understand it, derivatives are taken from rules and associated with the corresponding values. These derivatives are then evaluated and accumulated in numeric form.

To evaluate a derivative we need it’s arguments, meaning that the intermediary results will have to be kept from the evaluation of the overall function, if we want to accumulate the derivatives from the outermost function (in ML typically the loss function) and inwards.

A simple explanation of the differences between Algorithmic Differentiation (A), Numeric Differentiation (N) and Symbolic Differentiation (S) could be shown in a figure:

```julia
                                                        ┏━━━━━┳━━━━━━━┓
                                                        ┃ sym ┃ num ┃
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╋━━━━━╋━━━━━━━┫
┃differentiating primitive parts ("leaves in the graph")┃ A,S ┃ (A),N ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╋━━━━━╋━━━━━━━┫
┃accumulating the derivative ("non-leaves") ┃ S ┃ A,N ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━┻━━━━━━━┛

```

Here `sym` and `num` refer to if the part in question is calculated symbolically or numerically. I surmise that Algorithmic Differentiation could potentially revert to numerical evaluation of “unfamiliar” nodes and that is why I put an A in parenthesis in the upper right corner.

Apart perhaps from the minutest semantic details, I would like to know if this is a correct description.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [March 26, 2023, 2:17pm UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/9 "2023-03-26T14:17:40Z")

</div>

Links to introductory or survey journal articles:

[https://www.jstor.org/stable/24103956](https://www.jstor.org/stable/24103956)

> **[17-468.pdf](https://www.jmlr.org/papers/volume18/17-468/17-468.pdf)**
>
> 591.62 KB

---

<div class="post-metadata">

**Author:** ![Euhan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/euhan/32/36548_2.png) [@Euhan](https://discourse.julialang.org/u/Euhan)\
**Post date:** [March 28, 2023, 10:20am UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/10 "2023-03-28T10:20:41Z")

</div>

Thanks again for all the references.

I have above given a short explanation, based on what I have learned from said references. What I would like to know is:

Is this explanation correct? If not, what is wrong?

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 28, 2023, 11:49am UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/11 "2023-03-28T11:49:44Z")

</div>

> [@Euhan](#):
>
> I surmise that Algorithmic Differentiation could potentially revert to numerical evaluation of “unfamiliar” nodes and that is why I put an A in parenthesis in the upper right corner.

No. Zygote and other AD systems never use finite differences.

Essentially, they accumulate vector–Jacobian products, working backwards from outputs to inputs (for reverse-mode AD; or Jacobian–vector products working from input to output for forward mode). These products are computed by an equivalent of symbolic differentiation for individual computational steps, expressed in low-level compiler building blocks rather than in high-level symbolic expressions/code. If it hits a function call that it cannot analyze (e.g. foreign function calls, mutation, …) then it fails with an error, and you need to supply a manual vector–Jacobian product (an `rrule` or “pullback” via [ChainRules.jl](https://github.com/JuliaDiff/ChainRules.jl) for Zygote) for that step.

---

<div class="post-metadata">

**Author:** ![Euhan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/euhan/32/36548_2.png) [@Euhan](https://discourse.julialang.org/u/Euhan)\
**Post date:** [March 29, 2023, 10:42am UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/12 "2023-03-29T10:42:13Z")

</div>

So, I conclude that my description was correct. To clarify: I only stated that I did not guarantee that numerical derivation couldn’t be used in _some_ algorithmic derivation system. Also, I can no longer edit the post in question.

Also, If I get to write my own derivation rules for unknown functions, what’s to say that I don’t write a rule like \frac{df}{dx}=\frac{f(x+\delta)-f(x)}{\delta}?

Is there any non-semantic reason that an algorithmic derivation system couldn’t use a numeric derivative for a certain group of primitives?

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 29, 2023, 10:52am UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/13 "2023-03-29T10:52:26Z")

</div>

> [@Euhan](#):
>
> Also, If I get to write my own derivation rules for unknown functions, what’s to say that I don’t write a rule like \frac{df}{dx}=\frac{f(x+\delta)-f(x)}{\delta}dfdx=f(x+δ)−f(x)δ\frac{df}{dx}=\frac{f(x+\delta)-f(x)}{\delta}?

Sure, you could do that. You could also write an incorrect derivative rule if you want. Or your derivative rule could query ChatGPT. Or your function could send an email to your high-school calculus teacher and wait until it receives a response. Code can do lots of things.

> [@Euhan](#):
>
> Is there any non-semantic reason that an algorithmic derivation system couldn’t use a numeric derivative for a certain group of primitives?

There’s nothing physically preventing it. Most people would consider that a bug in an AD system, though.

(A finite difference like that would be noticeable, even without looking inside the code, for having the wrong performance scaling — its cost scales proportional to the number of inputs, whereas reverse-mode AD scales proportional to the number of outputs — as well as unacceptably large numerical errors.)

---

<div class="post-metadata">

**Author:** ![Euhan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/euhan/32/36548_2.png) [@Euhan](https://discourse.julialang.org/u/Euhan)\
**Post date:** [March 29, 2023, 3:45pm UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/14 "2023-03-29T15:45:10Z")

</div>

Finite differences are clearly good enough in some application and perhaps even the only alternative. Should it then be understood that people doing that would have nothing to gain from introducing rule based derivatives in those places where it’s possible?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [March 29, 2023, 3:50pm UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/15 "2023-03-29T15:50:47Z")

</div>

you can very often at the very least use dual numbers which are more numerically robust and typically give you higher accuracy.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 29, 2023, 7:58pm UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/16 "2023-03-29T19:58:02Z")

</div>

> [@Euhan](#):
>
> Should it then be understood that people doing that would have nothing to gain from introducing rule based derivatives

Only if you consider accuracy and scalability to be nothing. Which maybe they are, if finite differences are good enough and you have other concerns to focus on.

---

<div class="post-metadata">

**Author:** ![Euhan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/euhan/32/36548_2.png) [@Euhan](https://discourse.julialang.org/u/Euhan)\
**Post date:** [March 30, 2023, 11:20pm UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/17 "2023-03-30T23:20:15Z")

</div>

So, if and when they do this they have implemented an algorithmic derivation system partially using finite differences.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [March 31, 2023, 5:10am UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/18 "2023-03-31T05:10:47Z")

</div>

They would then have made a differentiation system that mixes AD and finite differences.

Seems like you are now mainly engaging in semantic bickering, in order to score some far-fetched point. If you think a hybrid system would be useful, then that’s fine, but there’s no reason to insist on it being fully AD.

---

<div class="post-metadata">

**Author:** ![Euhan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/euhan/32/36548_2.png) [@Euhan](https://discourse.julialang.org/u/Euhan)\
**Post date:** [March 31, 2023, 10:43am UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/19 "2023-03-31T10:43:55Z")

</div>

That is a correct analysis (although I don’t exclude the possibility that a hybrid differentiation system would be useful). The point, the importance of which you so rightly belittle, is that I asked for a **non-semantic** reason.

The bickering bit could possibly be connected to a certain annoyance that nobody could write the one or two sentences needed to convey the information in my table (possibly without any parenthetical letter), that would have been helpful to understanding both what Zygote does and what the distinction between different differentiation strategies is.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [March 31, 2023, 2:18pm UTC](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528/20 "2023-03-31T14:18:37Z")

</div>

Frankly, I just found the table more confusing because it wasn’t clear what each of the columns/rows/cells meant. In the spirit of a one-liner response though, I remember someone (maybe @ChrisRackauckas?) summarizing it as “AD is like symbolic differentiation, just with `=` [instead of deep nested expression trees]”. Probably butchering the quote, but should clarify that finite differences doesn’t even enter the picture here.

[Next page](https://discourse.julialang.org/t/does-zygote-differentiate-symbolically/96528.md?page=2)