There seems to be a plethora of AD frameworks in Julia. I’m curious what the state of the ecosystem looks like currently. I’m aware of the following packages: FowardDiff ForwardDiff2 Nabla Tracker Yota Zygote ReverseDiff AutoGrad NiLang ModelingToolkit XGrad Calculus FiniteDifferences FiniteDiff T…

[image] oxinabox: Finite differencing is effectively forward mode AD I don’t think that FD should be confused with AD.

[image] Tamas_Papp: I don’t think that FD should be confused with AD. It’s the same algorithm, you just store the perturbations in a different dimension.

[image] oxinabox: Finite differencing is effectively forward mode AD but less accurate. Super super robust though. This surprised me greatly. Firstly, I always heard they were fundamentally different; and secondly, I thought finite differences were very vulnerable to to noise and sensitive to…

Yes, I am aware of the theoretical connection, but for people new to the concept approaching it like this may be confusing, especially as FD requires close attention to a host of numerical issues that do not afflict AD.

[image] DNF: This surprised me greatly. Firstly, I always heard they were fundamentally different; and secondly, I thought finite differences were very vulnerable to to noise and sensitive to stepsize. They aren’t necessarily. Forward mode and reverse mode are fundamentally different: they ha…

Another practical difference in FD vs Forward mode is that FD can treat a function completely as a black-box requiring only function evaluation, whereas Forward AD will need to decompose the function into primitives with known derivative rules. So with FD you might even differentiate through some ex…

I’m interested in knowing how trustworthy the results from Zygote/Flux and Yota currently are. Zygote seems to already be used in Flux, but in its docs there is “At least, that’s the idea. We’re still in beta so expect some adventures”. I can’t really use it if there’s a chance that some bug is go…

[image] joshpritsker: some bug is going to make my results completely off if you depend it on rocket launching, don’t, cuz numerical stuff always go wrong and it’s impossible to guarantee otherwise. other than that, it’s pretty “accurate” but of course edge cases exist, but then again, don’t…

[image] joshpritsker: I’m interested in knowing how trustworthy the results from Zygote/Flux and Yota currently are. Jerry summarized it pretty well. Currently Yota passes all its 140 tests (+ a bunch of tests in Lilith.jl which further challenges Yota’s autodiff), but the universe of possibl…

As a rule, if you’re at all concerned about the correctness of the gradients you’re getting from an AD tool, you should test it using finite differencing eg. with FiniteDifferences.jl

State of automatic differentiation in Julia

Specific Domains Machine Learning

Topic		Replies	Views
[blog post] Implement your own AD with Julia in ONE day Community blog-post	33	4522	November 3, 2018
What lessons could Julia's autodiff ecosystem learn from Stan's TinyGrad? Machine Learning	45	5034	September 26, 2025
Comparison of automatic differentiation tools from 2016 still accurate? Numerics differentiation	41	6082	August 16, 2018
Is it possible to do Nested AD ~elegantly~ in Julia? (PINNs) General Usage machine-learning	43	3585	September 27, 2024
Automatic Differentiation (AD) in Julia vs. Python (or PyTorch) Machine Learning autodiff	14	2521	January 16, 2025

State of automatic differentiation in Julia

Related topics