# What's the state of Automatic Differentiation in Julia January 2023?

**URL:** <https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473>\
**Category:** General Usage\
**Tags:** autodiff\
**Created:** [January 3, 2023, 11:24pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473 "2023-01-03T23:24:37Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [May 6, 2023, 4:45pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/21 "2023-05-06T16:45:28Z")

</div>

> [@ChrisRackauckas](#):
>
> ```julia
> @show bx - bx2
> 
> #=
> 2-element Vector{Float64}:
> -9.992007221626409e-16
> -1.7763568394002505e-15
> =#
> 
> ```

On my M1 and Julia 1.9 rc3 with Enzyme#main, I am getting a difference between Enzyme and Zygote gradients of

```julia
2-element Vector{Float64}:
 0.15407736749039058
 0.22841827115822566

```

by running the same code above.

Edit: also `] test Enzyme` causes a segfault. Is the M1 not supported by Enzyme?

---

<div class="post-metadata">

**Author:** ![wsmoses](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsmoses/32/26497_2.png) [@wsmoses](https://discourse.julialang.org/u/wsmoses)\
**Post date:** [May 7, 2023, 2:00am UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/22 "2023-05-07T02:00:05Z")

</div>

Like I said in the other thread ([Use Enzyme in flux - #7 by wsmoses](https://discourse.julialang.org/t/use-enzyme-in-flux/98352/7)) the code Chris provided is not a good way to call it because of the type unstable closure capture of dudt, p, and st. These should be passed explicitly as Const parameters for performance and other reasons.

M1 indeed is not officially supported, partially because we don’t have a dev machine or CI for it. If you can isolate and post which test fails on GitHub we can try to look into it.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 7, 2023, 2:14am UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/23 "2023-05-07T02:14:04Z")

</div>

I can concur that it’s giving something odd on my M2.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [May 7, 2023, 3:24pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/24 "2023-05-07T15:24:21Z")

</div>

Thanks William. I tried Windows and got the exact same difference in gradients. So it’s not an M1 thing. Enzyme tests pass on Windows. I suppose that example is not supposed to work at all?

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [May 7, 2023, 5:29pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/25 "2023-05-07T17:29:01Z")

</div>

On my Intel CPU, the example just throws errors.

```julia
julia> Enzyme.autodiff(Reverse, f, Duplicated(x, bx), Duplicated(y, ones(2)))
warning: didn't implement memmove, using memcpy as fallback which can result in errors
ERROR: Enzyme execution failed.
Enzyme: not yet implemented in reverse mode, jl_getfield
Stacktrace:
 [1] getindex
   @ ./tuple.jl:29
 [2] getindex
   @ ./tuple.jl:0

Stacktrace:
 [1] throwerr(cstr::Cstring)
   @ Enzyme.Compiler ~/.julia/packages/Enzyme/YBQJk/src/compiler.jl:2536
 [2] macro expansion
   @ ~/.julia/packages/Enzyme/YBQJk/src/compiler.jl:8646 [inlined]
 [3] enzyme_call
   @ ~/.julia/packages/Enzyme/YBQJk/src/compiler.jl:8338 [inlined]
 [4] CombinedAdjointThunk
   @ ~/.julia/packages/Enzyme/YBQJk/src/compiler.jl:8301 [inlined]
 [5] autodiff
   @ ~/.julia/packages/Enzyme/YBQJk/src/Enzyme.jl:205 [inlined]
 [6] autodiff
   @ ~/.julia/packages/Enzyme/YBQJk/src/Enzyme.jl:228 [inlined]
 [7] autodiff(::EnzymeCore.ReverseMode{false}, ::typeof(f), ::Duplicated{Vector{Float64}}, ::Duplicated{Vector{Float64}})
   @ Enzyme ~/.julia/packages/Enzyme/YBQJk/src/Enzyme.jl:214
 [8] top-level scope
   @ REPL[12]:1

```

This is common across Julia 1.8.5, 1.9-rc3, and master.  
This was with Enzyme 0.11.1. I also tried 0.11.0 on 1.9-rc3.

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [May 7, 2023, 5:41pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/26 "2023-05-07T17:41:13Z")

</div>

Well @ChrisRackauckas did say he was using the unreleased main branch of Enzyme, not any of the registered versions.

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [May 7, 2023, 5:47pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/27 "2023-05-07T17:47:19Z")

</div>

🤦‍♂️  
Now I get

```julia
julia> @show bx - bx2
bx - bx2 = [0.1540773674903897, 0.22841827115822566]
2-element Vector{Float64}:
 0.1540773674903897
 0.22841827115822566

julia> versioninfo()
Julia Version 1.9.0-rc3
Commit 1853b90328 (2023-04-26 15:51 UTC)
Platform Info:
  OS: Linux (x86_64-generic-linux)
  CPU: 28 × Intel(R) Core(TM) i9-9940X CPU @ 3.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake-avx512)

```

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [May 7, 2023, 5:52pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/28 "2023-05-07T17:52:04Z")

</div>

@ChrisRackauckas we seem to all be getting the exact same non-zero difference on MacOS, Windows and Linux. Which OS did you use to run the above example 😅?

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [May 7, 2023, 7:05pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/29 "2023-05-07T19:05:14Z")

</div>

> [@wsmoses](#):
>
> M1 indeed is not officially supported, partially because we don’t have a dev machine or CI for it

For CI I’m happy to help you set up Cirrus on M1, if that helps

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 7, 2023, 10:05pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/30 "2023-05-07T22:05:46Z")

</div>

Peculiar. It was on Windows, and now I cannot recreate it. I do not know the manifest of the time, though I do know that it was in a clean REPL. It is quite mysterious to me 😅. Maybe a difference in Julia version?

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [May 8, 2023, 5:04am UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/31 "2023-05-08T05:04:05Z")

</div>

> [@ChrisRackauckas](#):
>
> Maybe a difference in Julia version?

Probably not. I tested on Julia 1.9 rc3. Chris Elrod tests 3 Julia versions. It’s most likely dependencies but I am guessing 🤷

Anyways, I think Enzyme will be well served by a simpler AbstractDifferentiation API for non-mutating functions. Probably worth giving it another go soon-ish.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 8, 2023, 8:18am UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/32 "2023-05-08T08:18:03Z")

</div>

> [@wsmoses](#):
>
> intend to fix these, but in the interim, I’ve been chasing other features (like GC/type unstability, and also defending my thesis on Monday 😛 ).
> 
> @Tamas_Papp, perhaps just turn that flag on by default?

Again, I think Enzyme is really promising, blazing fast when it works, and probably the best way forward. I really appreciate all the great work developers have put into it. It is just that, at the moment, it is not stable enough for those who are not prepared to invest extra effort in checking and debugging (again, for nontrivial examples).

For my particular issue, yes, the workaround works. But I ended up rewriting the function in a C-style approach: instead of higher order functions, just loop over stuff and increment a float. This works fine.

My other meta-gripe with Enzyme is that it is tricky to understand the error messages. Whenever something goes wrong, hundreds of lines of IR are dumped on the user. Again, I understand that mapping that back to Julia code is tricky, but in this form it is difficult to isolate errors and produce MWEs.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [May 8, 2023, 9:38am UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/33 "2023-05-08T09:38:43Z")

</div>

> [@mohamed82008](#):
>
> Anyways, I think Enzyme will be well served by a simpler AbstractDifferentiation API for non-mutating functions. Probably worth giving it another go soon-ish.

> <https://github.com/JuliaDiff/AbstractDifferentiation.jl/pull/85>
>
> This is a draft PR because Enzyme gives incorrect gradients in the tests. I susp…ect that's because of global captures related to this thread https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/21.

I think the global capture issue is causing wrong gradients though so my initial tests didn’t pass.

Edit: no segfaults though which is an improvement from last time I played with Enzyme for AbstractDifferentiation

---

<div class="post-metadata">

**Author:** ![wsmoses](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsmoses/32/26497_2.png) [@wsmoses](https://discourse.julialang.org/u/wsmoses)\
**Post date:** [May 8, 2023, 4:04pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/34 "2023-05-08T16:04:59Z")

</div>

I’ve landed some code that will automatically throw an error, saying to enable runtimeActivity if it detects a scenario where it is needed. Hopefully that should at least partially abate issues (all the activity bugs on the Enzyme.jl tracker hit it).

The longer error messages involve a combination of issues. The biggest one at the moment is an issue on Julia proper that we’ve been waiting for someone to look at since March =/ ([Frame registration for foreign JITs · Issue #49056 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/49056)).

@ChrisRackauckas am not sure, but our unreleased main branch is an unreleased main branch for a reason. It usually has a lot more fixes, but it also means we can break it (like I did over the weekend to update the ABI, which awaited a jll bump). Not sure if that’s what you ran into (I never tested your code myself).

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 9, 2023, 12:09am UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/35 "2023-05-09T00:09:44Z")

</div>

🤦‍♂️ I had the bug report code in the same script, so it had runtime activity checking turned on from that part. The following is what works:

```julia
using Enzyme
Enzyme.API.runtimeActivity!(true)

x = [2.0, 2.0]
bx = [0.0, 0.0]
y = [0.0,0.0]

using ComponentArrays, Lux, Random

rng = Random.default_rng()
Random.seed!(rng,100)
dudt2 = Lux.Chain(x -> x.^3,
                  Lux.Dense(2, 50, tanh),
                  Lux.Dense(50, 2))
p, st = Lux.setup(rng, dudt2)

function f(x::Array{Float64}, y::Array{Float64})
    y .= dudt2(x, p, st)[1]
    return nothing
end

Enzyme.autodiff(Reverse, f, Duplicated(x, bx), Duplicated(y, ones(2)))

function f2(x::Array{Float64})
    dudt2(x, p, st)[1]
end

using Zygote
bx2 = Zygote.pullback(f2, x)[2](ones(2))[1]

@show bx - bx2

```

The difference is `Enzyme.API.runtimeActivity!(true)`, so yeah @wsmoses chalk this one up to another runtimeActivity check.

> [@wsmoses](#):
>
> The biggest one at the moment is an issue on Julia proper that we’ve been waiting for someone to look at since March =/ ([Frame registration for foreign JITs · Issue #49056 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/49056)).

I talked with Jeff and Gabriel about this today. We were talking about a v1.10 milestone.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 15, 2023, 1:16pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/37 "2023-05-15T13:16:20Z")

</div>

In the past few days I have been trying to make Enzyme work on a medium-sized log posterior calculation. Ran into what turned out to be 2 (or 3?) separate issues, and got a lot of help from @wsmoses with isolating them as an MWE, at which point they were promptly fixed.

Just want to say **thank you**! And recommend Enzyme for everyone who is looking for a fast reverse-mode AD solution. Understandably the library is maturing and there are bugs, but they can be tracked down and the support from the authors is outstanding.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 15, 2023, 1:21pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/38 "2023-05-15T13:21:23Z")

</div>

> [@Tamas\_Papp](#):
>
> Just want to say **thank you**! And recommend Enzyme for everyone who is looking for a fast reverse-mode AD solution. Understandably the library is maturing and there are bugs, but they can be tracked down and the support from the authors is outstanding.

👍 that’s what I was trying to convey. Is it perfect right now? No. But it’s definitely on the right path in terms of supporting the full language, and it’s moving pretty fast. Check your answers of course since it’s still early stages, but with v0.11 it’s really looking on track to where it needs to be.

---

<div class="post-metadata">

**Author:** ![Maysam\_Gholampour](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maysam_gholampour/32/218184_2.png) [@Maysam\_Gholampour](https://discourse.julialang.org/u/Maysam_Gholampour)\
**Post date:** [June 20, 2024, 5:14pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/39 "2024-06-20T17:14:43Z")

</div>

Thanks a lot. I received error when I tried to use the enzyme for gpu case in this example. Does enzyme support gpu?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [June 21, 2024, 10:56pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/40 "2024-06-21T22:56:36Z")

</div>

It got added. Enzyme now supports standard GPU calls which includes broadcast and CuBLAS overloads, along with NNLib overloads (so standard neural network kernels) where the last is via the Enzyme rules system which was added I believe after this thread. GPU kernels of Julia code IIUC work as well. But this is about at the end of current Enzyme support so you’ll need the latest versions and YMMV. I wouldn’t call Enzyme with GPUs “solved” but it’s getting close.

---

<div class="post-metadata">

**Author:** ![wsmoses](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wsmoses/32/26497_2.png) [@wsmoses](https://discourse.julialang.org/u/wsmoses)\
**Post date:** [June 21, 2024, 11:09pm UTC](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473/41 "2024-06-21T23:09:22Z")

</div>

> [@ChrisRackauckas](#):
>
> supports standard GPU calls which includes broadcast and CuBLAS overloads, along with NNLib overloads (so standard neural network kernels) where the last is via the Enzyme rules system which was added I believe after this thread. GPU kernels of Julia code IIUC work as well. But this is about at the end of current Enzyme support so you’ll need the latest versions and YMMV. I wou

The broadcast kernel default support only has forward mode landed ([Add Enzyme Forward mode custom rule by wsmoses · Pull Request #1869 · JuliaGPU/CUDA.jl · GitHub](https://github.com/JuliaGPU/CUDA.jl/pull/1869)). A PR for reverse mode is open to CUDA.jl here: [Enzyme: reverse mode kernels by wsmoses · Pull Request #2422 · JuliaGPU/CUDA.jl · GitHub](https://github.com/JuliaGPU/CUDA.jl/pull/2422) but it hit a seemingly unrelated error in GPUCompiler.jl and/or CUDA.jl ([GPUCompiler emit\_exception has wrong number o args · Issue #2423 · JuliaGPU/CUDA.jl · GitHub](https://github.com/JuliaGPU/CUDA.jl/issues/2423))

cc @maleadt @vchuravy

[Previous page](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473.md?page=1)

[Next page](https://discourse.julialang.org/t/whats-the-state-of-automatic-differentiation-in-julia-january-2023/92473.md?page=3)