# Optimisers.jl cannot be used with Zygote.jl's implicit gradients

**URL:** <https://discourse.julialang.org/t/optimisers-jl-cannot-be-used-with-zygote-jls-implicit-gradients/102522>\
**Category:** General Usage\
**Created:** [August 5, 2023, 8:44pm UTC](https://discourse.julialang.org/t/optimisers-jl-cannot-be-used-with-zygote-jls-implicit-gradients/102522 "2023-08-05T20:44:33Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![George\_Stepaniants](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/george_stepaniants/32/47230_2.png) [@George\_Stepaniants](https://discourse.julialang.org/u/George_Stepaniants)\
**Post date:** [August 5, 2023, 8:44pm UTC](https://discourse.julialang.org/t/optimisers-jl-cannot-be-used-with-zygote-jls-implicit-gradients/102522/1 "2023-08-05T20:44:33Z")

</div>

I’m having the following issue with using Zygote with Optimisers.jl. Here is an MRE:

```julia
using Flux, Zygote

m = Chain(Dense(10, 5, relu), Dense(5, 2))

gs = gradient(() -> sum(m(x)), params(m))

opt = ADAM();

Flux.Optimise.update!(opt, params(m), gs)

```

which returns the error

```julia
ERROR: Optimisers.jl cannot be used with Zygote.jl's implicit gradients, `Params` & `Grads`
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] base(dx::Zygote.Grads)
   @ Flux ~/.julia/packages/Flux/n3cOc/src/Flux.jl:20
 [3] (::Optimisers.var"#13#15"{Params{Zygote.Buffer{Any, Vector{Any}}}})(x̄::Zygote.Grads)
   @ Optimisers ~/.julia/packages/Optimisers/1x8gl/src/interface.jl:112
 [4] map
   @ ./tuple.jl:273 [inlined]
 [5] _grads!(dict::IdDict{Optimisers.Leaf, Any}, tree::Optimisers.Adam{Float32}, x::Params{Zygote.Buffer{Any, Vector{Any}}}, x̄s::Zygote.Grads)
   @ Optimisers ~/.julia/packages/Optimisers/1x8gl/src/interface.jl:112
 [6] update!(::Optimisers.Adam{Float32}, ::Params{Zygote.Buffer{Any, Vector{Any}}}, ::Zygote.Grads)
   @ Optimisers ~/.julia/packages/Optimisers/1x8gl/src/interface.jl:70
 [7] top-level scope
   @ REPL[106]:1

```

---

<div class="post-metadata">

**Author:** ![bertschi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bertschi/32/33462_2.png) [@bertschi](https://discourse.julialang.org/u/bertschi)\
**Post date:** [August 5, 2023, 9:21pm UTC](https://discourse.julialang.org/t/optimisers-jl-cannot-be-used-with-zygote-jls-implicit-gradients/102522/2 "2023-08-05T21:21:34Z")

</div>

`Optimisers` seems to no longer support implicit gradients. The [docs](https://fluxml.ai/Optimisers.jl/dev/#Usage-with-%5BFlux.jl%5D(https://github.com/FluxML/Flux.jl)) show how to use Flux with explicit gradients instead. For your example, the following should work:

```julia
gs, _ = gradient((model, inp) -> sum(model(inp)), m, x)

opt = Optimisers.ADAM()
opt_state = Optimisers.setup(opt, m)
# Do an update step
opt_state, m = Optimisers.update(opt_state, m, gs)

```

---

<div class="post-metadata">

**Author:** ![George\_Stepaniants](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/george_stepaniants/32/47230_2.png) [@George\_Stepaniants](https://discourse.julialang.org/u/George_Stepaniants)\
**Post date:** [August 5, 2023, 9:30pm UTC](https://discourse.julialang.org/t/optimisers-jl-cannot-be-used-with-zygote-jls-implicit-gradients/102522/3 "2023-08-05T21:30:21Z")

</div>

I see, then why are there so many examples of implicit gradients in the Flux documentation if optimisation with such gradients is no longer supported?

[https://fluxml.ai/Flux.jl/stable/models/basics/#Building-Simple-Models](https://fluxml.ai/Flux.jl/stable/models/basics/#Building-Simple-Models)  
[https://fluxml.ai/Flux.jl/stable/training/training/#Model-Gradients](https://fluxml.ai/Flux.jl/stable/training/training/#Model-Gradients)

---

<div class="post-metadata">

**Author:** ![bertschi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bertschi/32/33462_2.png) [@bertschi](https://discourse.julialang.org/u/bertschi)\
**Post date:** [August 5, 2023, 11:23pm UTC](https://discourse.julialang.org/t/optimisers-jl-cannot-be-used-with-zygote-jls-implicit-gradients/102522/4 "2023-08-05T23:23:11Z")

</div>

Don’t know when this will be updated. On the other hand, the optimisers from `Flux` itself, i.e., not loading `Optimisers`, still work for me with implicit gradients:

```julia
gs = Zygote.gradient(() -> sum(m(x)), Flux.params(m))
opt = Flux.Optimise.ADAM();
Flux.Optimise.update!(opt, Flux.params(m), gs)

```