# Zygote debugging

**URL:** <https://discourse.julialang.org/t/zygote-debugging/28423>\
**Category:** General Usage\
**Created:** [September 5, 2019, 11:17am UTC](https://discourse.julialang.org/t/zygote-debugging/28423 "2019-09-05T11:17:03Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![HenrikM](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrikm/32/17975_2.png) [@HenrikM](https://discourse.julialang.org/u/HenrikM)\
**Post date:** [September 5, 2019, 11:17am UTC](https://discourse.julialang.org/t/zygote-debugging/28423/1 "2019-09-05T11:17:03Z")

</div>

How to debug Zygote?

Currently, when running  
`gradient(fun, ω)`  
I get the following error message below. What does it mean? How should I read it? Line 9 below refers to one of my functions, but the error is probably not on that line. How should I start chopping up my code to produce a manageable small example? What kind of checklist should I use when approaching this issue?

```julia
MethodError: no method matching zero(::Tuple{Float64,Float64})
Closest candidates are:
  zero(!Matched::Type{LibGit2.GitHash}) at /usr/share/julia/stdlib/v1.1/LibGit2/src/oid.jl:220
  zero(!Matched::Type{Pkg.Resolve.VersionWeights.VersionWeight}) at /usr/share/julia/stdlib/v1.1/Pkg/src/resolve/VersionWeights.jl:19
  zero(!Matched::Type{Pkg.Resolve.MaxSum.FieldValues.FieldValue}) at /usr/share/julia/stdlib/v1.1/Pkg/src/resolve/FieldValues.jl:44
  ...

Stacktrace:
 [1] (::getfield(Zygote, Symbol("##90#91")){Bool})(::Tuple{Float64,Float64}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:40
 [2] (::getfield(Zygote, Symbol("##154#back#92")){getfield(Zygote, Symbol("##90#91")){Bool}})(::Tuple{Float64,Float64}) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [3] add12 at ./twiceprecision.jl:84 [inlined]
 [4] (::typeof(∂(Base.add12)))(::Tuple{Float64,Float64}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [5] unsafe_getindex at ./twiceprecision.jl:450 [inlined]
 [6] (::typeof(∂(Base.unsafe_getindex)))(::Float64) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [7] getindex at ./range.jl:639 [inlined]
 [8] (::typeof(∂(getindex)))(::Float64) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [9] R2 at /Notebooks/backprop.jl:121 [inlined]
 [10] (::typeof(∂(Main.Backprop.R2)))(::Tuple{Array{Float64,2},Nothing,Nothing}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [11] (::typeof(∂(getfield(Main, Symbol("##3#4"))())))(::Float64) at ./In[4]:1
 [12] (::getfield(Zygote, Symbol("##32#33")){typeof(∂(getfield(Main, Symbol("##3#4"))()))})(::Float64) at /julia/packages/Zygote/bdE6T/src/compiler/interface.jl:38
 [13] gradient(::Function, ::Array{Float64,2}) at /julia/packages/Zygote/bdE6T/src/compiler/interface.jl:47
 [14] top-level scope at In[5]:1

```

---

<div class="post-metadata">

**Author:** ![HenrikM](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrikm/32/17975_2.png) [@HenrikM](https://discourse.julialang.org/u/HenrikM)\
**Post date:** [September 5, 2019, 12:09pm UTC](https://discourse.julialang.org/t/zygote-debugging/28423/2 "2019-09-05T12:09:07Z")

</div>

Below is the smallest possible function that exhibits what I think is the error. The error message is even more daunting, but at least the top row is clear: “Mutating arrays is not supported”. The recursion was my attempt at not mutating arrays, but apparently it does not work that way.

```julia
using Pkg
pkg"activate ."

using Zygote

function S1(x)
    d = length(x)

    t = x[1:1]
    
    for idx = 2:d
        t = recu(t, x[idx])
    end

    sum(t)
end

function recu(t, x)
    [t x]
end

x = collect(1.0:10)
S1(x)
gradient(S1, x)

```

```julia
ERROR: LoadError: Mutating arrays is not supported
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] (::getfield(Zygote, Symbol("##794#795")))(::Nothing) at /julia/packages/Zygote/bdE6T/src/lib/array.jl:37
 [3] (::getfield(Zygote, Symbol("##2100#back#796")){getfield(Zygote, Symbol("##794#795"))})(::Nothing) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [4] fill! at ./multidimensional.jl:839 [inlined]
 [5] (::typeof(∂(fill!)))(::Nothing) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [6] __cat at ./abstractarray.jl:1412 [inlined]
 [7] (::typeof(∂(Base.__cat)))(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [8] (::getfield(Zygote, Symbol("##143#144")){typeof(∂(Base.__cat)),Tuple{Tuple{Nothing,Nothing,Nothing},Tuple{Nothing,Nothing}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [9] (::getfield(Zygote, Symbol("##274#back#145")){getfield(Zygote, Symbol("##143#144")){typeof(∂(Base.__cat)),Tuple{Tuple{Nothing,Nothing,Nothing},Tuple{Nothing,Nothing}}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [10] _cat_t at ./abstractarray.jl:1392 [inlined]
 [11] (::getfield(Zygote, Symbol("##143#144")){typeof(∂(Base._cat_t)),Tuple{Tuple{Nothing,Nothing},Tuple{Nothing,Nothing}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [12] #143 at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48 [inlined]
 [13] (::getfield(Zygote, Symbol("##274#back#145")){getfield(Zygote, Symbol("##143#144")){typeof(∂(Base.#cat_t#103)),Tuple{Tuple{Nothing,Nothing,Nothing},Tuple{Nothing,Nothing}}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [14] #cat_t at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0 [inlined]
 [15] (::typeof(∂(getfield(Base, Symbol("#kw##cat_t"))())))(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [16] (::getfield(Zygote, Symbol("##143#144")){typeof(∂(getfield(Base, Symbol("#kw##cat_t"))())),Tuple{Tuple{Nothing,Nothing,Nothing},Tuple{Nothing,Nothing}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [17] #274#back at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48 [inlined]
 [18] _cat at ./abstractarray.jl:1382 [inlined]
 [19] (::typeof(∂(Base._cat)))(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [20] (::getfield(Zygote, Symbol("##143#144")){typeof(∂(Base._cat)),Tuple{Tuple{Nothing},Tuple{Nothing,Nothing}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [21] #143 at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48 [inlined]
 [22] (::getfield(Zygote, Symbol("##274#back#145")){getfield(Zygote, Symbol("##143#144")){typeof(∂(Base.#cat#104)),Tuple{Tuple{Nothing,Nothing},Tuple{Nothing,Nothing}}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [23] #cat at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0 [inlined]
 [24] (::typeof(∂(getfield(Base, Symbol("#kw##cat"))())))(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [25] (::getfield(Zygote, Symbol("##143#144")){typeof(∂(getfield(Base, Symbol("#kw##cat"))())),Tuple{Tuple{Nothing,Nothing},Tuple{Nothing,Nothing}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [26] #274#back at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48 [inlined]
 [27] hcat at ./abstractarray.jl:1491 [inlined]
 [28] (::typeof(∂(hcat)))(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [29] recu at quicktest.jl:19 [inlined]
 [30] (::typeof(∂(recu)))(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [31] S1 at quicktest.jl:12 [inlined]
 [32] (::typeof(∂(S1)))(::Float64) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [33] (::getfield(Zygote, Symbol("##32#33")){typeof(∂(S1))})(::Float64) at /julia/packages/Zygote/bdE6T/src/compiler/interface.jl:38
 [34] gradient(::Function, ::Array{Float64,1}) at /julia/packages/Zygote/bdE6T/src/compiler/interface.jl:47
 [35] top-level scope at none:0
 [36] include at ./boot.jl:326 [inlined]
 [37] include_relative(::Module, ::String) at ./loading.jl:1038
 [38] include(::Module, ::String) at ./sysimg.jl:29
 [39] exec_options(::Base.JLOptions) at ./client.jl:267
 [40] _start() at ./client.jl:436
in expression starting at quicktest.jl:24

```

---

<div class="post-metadata">

**Author:** ![MikeInnes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikeinnes/32/3656_2.png) [@MikeInnes](https://discourse.julialang.org/u/MikeInnes)\
**Post date:** [September 5, 2019, 12:45pm UTC](https://discourse.julialang.org/t/zygote-debugging/28423/3 "2019-09-05T12:45:31Z")

</div>

There are some notes on how to break down your code in the [docs](https://fluxml.ai/Zygote.jl/dev/internals/#Debugging-1).

Stacktraces are still a bit noisy, but in your second example we can read off that the error is coming from `recu` (lines 29 and 30) and there from `hcat` (27 and 28). The rest of the lines are functions `hcat` calls, all the way down to `fill!` (lines 4 and 5) which throws the mutation error.

The right solution would be to add a gradient for `hcat` (I thought we had one, but we might just be missing a method).

---

<div class="post-metadata">

**Author:** ![HenrikM](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrikm/32/17975_2.png) [@HenrikM](https://discourse.julialang.org/u/HenrikM)\
**Post date:** [September 5, 2019, 1:30pm UTC](https://discourse.julialang.org/t/zygote-debugging/28423/4 "2019-09-05T13:30:35Z")

</div>

Thank you for your reply!

Point [9] in the stacktrace in the first post is this:

```julia
[9] R2 at /Notebooks/backprop.jl:121 [inlined]

```

That line is

```julia
x, f, U = step2(x, f, U, t[k] - t[k-1], z[:, 1:k-1], z[:, k])

```

Using the values I get from the forward pass of R2, I can to this:

```julia
y, back = Zygote._forward(
    step2,
    x, f, U, t[k] - t[k-1], z[:, 1:k-1], z[:, k])

```

This works and produces

```julia
(([0.0 -0.0415986 -0.0851587; 0.0 -0.0101956 -0.0182404], [-1.20636 -1.26324 -1.32588; -0.295674 -0.233297 -0.178448], [1.0 0.998167 0.992443; 0.0 0.0605315 0.122508; 0.0 0.0 0.00697798]), ∂(Main.Backprop.step2))

```

But what is now the function `back`? If I understand the [Debugging](https://fluxml.ai/Zygote.jl/dev/internals/#Debugging-1) section correctly, I should be able to do

```julia
back(y[1], y[2], y[3])

```

That is, using the returned values in the returned function. But this results in

```julia
MethodError: no method matching (::typeof(∂(Main.Backprop.step2)))(::Array{Float64,2}, ::Array{Float64,2}, ::UpperTriangular{Float64,Array{Float64,2}})
Closest candidates are:
  Pullback(::Any) where T at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:19

Stacktrace:
 [1] top-level scope at In[56]:1

```

---

<div class="post-metadata">

**Author:** ![MikeInnes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikeinnes/32/3656_2.png) [@MikeInnes](https://discourse.julialang.org/u/MikeInnes)\
**Post date:** [September 5, 2019, 1:39pm UTC](https://discourse.julialang.org/t/zygote-debugging/28423/5 "2019-09-05T13:39:44Z")

</div>

`back` only takes one input, with the shape of `y`; so you don’t want to split `y` up that way. `back(y)` will probably work, as long as `y` is a tuple of simple types like arrays or numbers.

---

<div class="post-metadata">

**Author:** ![HenrikM](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrikm/32/17975_2.png) [@HenrikM](https://discourse.julialang.org/u/HenrikM)\
**Post date:** [September 5, 2019, 3:39pm UTC](https://discourse.julialang.org/t/zygote-debugging/28423/6 "2019-09-05T15:39:10Z")

</div>

continued…  
This is great! Now I can do `back(y)` and get the error message below. Line [14] gives me hope that by adding a gradient for `hvcat` it might work. Is this correct?

```julia
Mutating arrays is not supported

Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] (::getfield(Zygote, Symbol("##794#795")))(::Nothing) at /julia/packages/Zygote/bdE6T/src/lib/array.jl:37
 [3] (::getfield(Zygote, Symbol("##2100#back#796")){getfield(Zygote, Symbol("##794#795"))})(::Nothing) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [4] _typed_vcat at ./abstractarray.jl:1313 [inlined]
 [5] (::typeof(∂(Base._typed_vcat)))(::UpperTriangular{Float64,Array{Float64,2}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [6] typed_vcat at ./abstractarray.jl:1319 [inlined]
 [7] (::typeof(∂(Base.typed_vcat)))(::UpperTriangular{Float64,Array{Float64,2}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [8] (::getfield(Zygote, Symbol("##143#144")){typeof(∂(Base.typed_vcat)),Tuple{Tuple{Nothing},Int64}})(::UpperTriangular{Float64,Array{Float64,2}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [9] (::getfield(Zygote, Symbol("##274#back#145")){getfield(Zygote, Symbol("##143#144")){typeof(∂(Base.typed_vcat)),Tuple{Tuple{Nothing},Int64}}})(::UpperTriangular{Float64,Array{Float64,2}}) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [10] typed_hvcat at ./abstractarray.jl:1687 [inlined]
 [11] (::typeof(∂(Base.typed_hvcat)))(::UpperTriangular{Float64,Array{Float64,2}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [12] (::getfield(Zygote, Symbol("##143#144")){typeof(∂(Base.typed_hvcat)),Tuple{Tuple{Nothing,Nothing},NTuple{4,Nothing}}})(::UpperTriangular{Float64,Array{Float64,2}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [13] #274#back at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48 [inlined]
 [14] hvcat at ./abstractarray.jl:1662 [inlined]
 [15] step2 at /Notebooks/backprop.jl:147 [inlined]
 [16] (::typeof(∂(Main.Backprop.step2)))(::Tuple{Array{Float64,2},Array{Float64,2},UpperTriangular{Float64,Array{Float64,2}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [17] top-level scope at In[8]:1

```

---

<div class="post-metadata">

**Author:** ![HenrikM](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrikm/32/17975_2.png) [@HenrikM](https://discourse.julialang.org/u/HenrikM)\
**Post date:** [October 1, 2019, 6:23am UTC](https://discourse.julialang.org/t/zygote-debugging/28423/7 "2019-10-01T06:23:05Z")

</div>

In the end, Zygote could not handle the code and I had to write my own adjoint. Maybe I will take a look at Zygote in the future.