Zygote debugging

General Usage
1

How to debug Zygote?

Currently, when running
gradient(fun, Ο‰)
I get the following error message below. What does it mean? How should I read it? Line 9 below refers to one of my functions, but the error is probably not on that line. How should I start chopping up my code to produce a manageable small example? What kind of checklist should I use when approaching this issue?

MethodError: no method matching zero(::Tuple{Float64,Float64})
Closest candidates are:
  zero(!Matched::Type{LibGit2.GitHash}) at /usr/share/julia/stdlib/v1.1/LibGit2/src/oid.jl:220
  zero(!Matched::Type{Pkg.Resolve.VersionWeights.VersionWeight}) at /usr/share/julia/stdlib/v1.1/Pkg/src/resolve/VersionWeights.jl:19
  zero(!Matched::Type{Pkg.Resolve.MaxSum.FieldValues.FieldValue}) at /usr/share/julia/stdlib/v1.1/Pkg/src/resolve/FieldValues.jl:44
  ...

Stacktrace:
 [1] (::getfield(Zygote, Symbol("##90#91")){Bool})(::Tuple{Float64,Float64}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:40
 [2] (::getfield(Zygote, Symbol("##154#back#92")){getfield(Zygote, Symbol("##90#91")){Bool}})(::Tuple{Float64,Float64}) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [3] add12 at ./twiceprecision.jl:84 [inlined]
 [4] (::typeof(βˆ‚(Base.add12)))(::Tuple{Float64,Float64}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [5] unsafe_getindex at ./twiceprecision.jl:450 [inlined]
 [6] (::typeof(βˆ‚(Base.unsafe_getindex)))(::Float64) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [7] getindex at ./range.jl:639 [inlined]
 [8] (::typeof(βˆ‚(getindex)))(::Float64) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [9] R2 at /Notebooks/backprop.jl:121 [inlined]
 [10] (::typeof(βˆ‚(Main.Backprop.R2)))(::Tuple{Array{Float64,2},Nothing,Nothing}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [11] (::typeof(βˆ‚(getfield(Main, Symbol("##3#4"))())))(::Float64) at ./In[4]:1
 [12] (::getfield(Zygote, Symbol("##32#33")){typeof(βˆ‚(getfield(Main, Symbol("##3#4"))()))})(::Float64) at /julia/packages/Zygote/bdE6T/src/compiler/interface.jl:38
 [13] gradient(::Function, ::Array{Float64,2}) at /julia/packages/Zygote/bdE6T/src/compiler/interface.jl:47
 [14] top-level scope at In[5]:1
2

Below is the smallest possible function that exhibits what I think is the error. The error message is even more daunting, but at least the top row is clear: β€œMutating arrays is not supported”. The recursion was my attempt at not mutating arrays, but apparently it does not work that way.

using Pkg
pkg"activate ."

using Zygote

function S1(x)
    d = length(x)

    t = x[1:1]
    
    for idx = 2:d
        t = recu(t, x[idx])
    end

    sum(t)
end

function recu(t, x)
    [t x]
end

x = collect(1.0:10)
S1(x)
gradient(S1, x)

ERROR: LoadError: Mutating arrays is not supported
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] (::getfield(Zygote, Symbol("##794#795")))(::Nothing) at /julia/packages/Zygote/bdE6T/src/lib/array.jl:37
 [3] (::getfield(Zygote, Symbol("##2100#back#796")){getfield(Zygote, Symbol("##794#795"))})(::Nothing) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [4] fill! at ./multidimensional.jl:839 [inlined]
 [5] (::typeof(βˆ‚(fill!)))(::Nothing) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [6] __cat at ./abstractarray.jl:1412 [inlined]
 [7] (::typeof(βˆ‚(Base.__cat)))(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [8] (::getfield(Zygote, Symbol("##143#144")){typeof(βˆ‚(Base.__cat)),Tuple{Tuple{Nothing,Nothing,Nothing},Tuple{Nothing,Nothing}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [9] (::getfield(Zygote, Symbol("##274#back#145")){getfield(Zygote, Symbol("##143#144")){typeof(βˆ‚(Base.__cat)),Tuple{Tuple{Nothing,Nothing,Nothing},Tuple{Nothing,Nothing}}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [10] _cat_t at ./abstractarray.jl:1392 [inlined]
 [11] (::getfield(Zygote, Symbol("##143#144")){typeof(βˆ‚(Base._cat_t)),Tuple{Tuple{Nothing,Nothing},Tuple{Nothing,Nothing}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [12] #143 at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48 [inlined]
 [13] (::getfield(Zygote, Symbol("##274#back#145")){getfield(Zygote, Symbol("##143#144")){typeof(βˆ‚(Base.#cat_t#103)),Tuple{Tuple{Nothing,Nothing,Nothing},Tuple{Nothing,Nothing}}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [14] #cat_t at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0 [inlined]
 [15] (::typeof(βˆ‚(getfield(Base, Symbol("#kw##cat_t"))())))(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [16] (::getfield(Zygote, Symbol("##143#144")){typeof(βˆ‚(getfield(Base, Symbol("#kw##cat_t"))())),Tuple{Tuple{Nothing,Nothing,Nothing},Tuple{Nothing,Nothing}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [17] #274#back at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48 [inlined]
 [18] _cat at ./abstractarray.jl:1382 [inlined]
 [19] (::typeof(βˆ‚(Base._cat)))(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [20] (::getfield(Zygote, Symbol("##143#144")){typeof(βˆ‚(Base._cat)),Tuple{Tuple{Nothing},Tuple{Nothing,Nothing}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [21] #143 at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48 [inlined]
 [22] (::getfield(Zygote, Symbol("##274#back#145")){getfield(Zygote, Symbol("##143#144")){typeof(βˆ‚(Base.#cat#104)),Tuple{Tuple{Nothing,Nothing},Tuple{Nothing,Nothing}}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [23] #cat at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0 [inlined]
 [24] (::typeof(βˆ‚(getfield(Base, Symbol("#kw##cat"))())))(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [25] (::getfield(Zygote, Symbol("##143#144")){typeof(βˆ‚(getfield(Base, Symbol("#kw##cat"))())),Tuple{Tuple{Nothing,Nothing},Tuple{Nothing,Nothing}}})(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [26] #274#back at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48 [inlined]
 [27] hcat at ./abstractarray.jl:1491 [inlined]
 [28] (::typeof(βˆ‚(hcat)))(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [29] recu at quicktest.jl:19 [inlined]
 [30] (::typeof(βˆ‚(recu)))(::FillArrays.Fill{Float64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [31] S1 at quicktest.jl:12 [inlined]
 [32] (::typeof(βˆ‚(S1)))(::Float64) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [33] (::getfield(Zygote, Symbol("##32#33")){typeof(βˆ‚(S1))})(::Float64) at /julia/packages/Zygote/bdE6T/src/compiler/interface.jl:38
 [34] gradient(::Function, ::Array{Float64,1}) at /julia/packages/Zygote/bdE6T/src/compiler/interface.jl:47
 [35] top-level scope at none:0
 [36] include at ./boot.jl:326 [inlined]
 [37] include_relative(::Module, ::String) at ./loading.jl:1038
 [38] include(::Module, ::String) at ./sysimg.jl:29
 [39] exec_options(::Base.JLOptions) at ./client.jl:267
 [40] _start() at ./client.jl:436
in expression starting at quicktest.jl:24
3

There are some notes on how to break down your code in the docs.

Stacktraces are still a bit noisy, but in your second example we can read off that the error is coming from recu (lines 29 and 30) and there from hcat (27 and 28). The rest of the lines are functions hcat calls, all the way down to fill! (lines 4 and 5) which throws the mutation error.

The right solution would be to add a gradient for hcat (I thought we had one, but we might just be missing a method).

1 Like
4

Thank you for your reply!

Point [9] in the stacktrace in the first post is this:

[9] R2 at /Notebooks/backprop.jl:121 [inlined]

That line is

x, f, U = step2(x, f, U, t[k] - t[k-1], z[:, 1:k-1], z[:, k])

Using the values I get from the forward pass of R2, I can to this:

y, back = Zygote._forward(
    step2,
    x, f, U, t[k] - t[k-1], z[:, 1:k-1], z[:, k])

This works and produces

(([0.0 -0.0415986 -0.0851587; 0.0 -0.0101956 -0.0182404], [-1.20636 -1.26324 -1.32588; -0.295674 -0.233297 -0.178448], [1.0 0.998167 0.992443; 0.0 0.0605315 0.122508; 0.0 0.0 0.00697798]), βˆ‚(Main.Backprop.step2))

But what is now the function back? If I understand the Debugging section correctly, I should be able to do

back(y[1], y[2], y[3])

That is, using the returned values in the returned function. But this results in

MethodError: no method matching (::typeof(βˆ‚(Main.Backprop.step2)))(::Array{Float64,2}, ::Array{Float64,2}, ::UpperTriangular{Float64,Array{Float64,2}})
Closest candidates are:
  Pullback(::Any) where T at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:19

Stacktrace:
 [1] top-level scope at In[56]:1
5

back only takes one input, with the shape of y; so you don’t want to split y up that way. back(y) will probably work, as long as y is a tuple of simple types like arrays or numbers.

6

continued…
This is great! Now I can do back(y) and get the error message below. Line [14] gives me hope that by adding a gradient for hvcat it might work. Is this correct?

Mutating arrays is not supported

Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] (::getfield(Zygote, Symbol("##794#795")))(::Nothing) at /julia/packages/Zygote/bdE6T/src/lib/array.jl:37
 [3] (::getfield(Zygote, Symbol("##2100#back#796")){getfield(Zygote, Symbol("##794#795"))})(::Nothing) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [4] _typed_vcat at ./abstractarray.jl:1313 [inlined]
 [5] (::typeof(βˆ‚(Base._typed_vcat)))(::UpperTriangular{Float64,Array{Float64,2}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [6] typed_vcat at ./abstractarray.jl:1319 [inlined]
 [7] (::typeof(βˆ‚(Base.typed_vcat)))(::UpperTriangular{Float64,Array{Float64,2}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [8] (::getfield(Zygote, Symbol("##143#144")){typeof(βˆ‚(Base.typed_vcat)),Tuple{Tuple{Nothing},Int64}})(::UpperTriangular{Float64,Array{Float64,2}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [9] (::getfield(Zygote, Symbol("##274#back#145")){getfield(Zygote, Symbol("##143#144")){typeof(βˆ‚(Base.typed_vcat)),Tuple{Tuple{Nothing},Int64}}})(::UpperTriangular{Float64,Array{Float64,2}}) at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48
 [10] typed_hvcat at ./abstractarray.jl:1687 [inlined]
 [11] (::typeof(βˆ‚(Base.typed_hvcat)))(::UpperTriangular{Float64,Array{Float64,2}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [12] (::getfield(Zygote, Symbol("##143#144")){typeof(βˆ‚(Base.typed_hvcat)),Tuple{Tuple{Nothing,Nothing},NTuple{4,Nothing}}})(::UpperTriangular{Float64,Array{Float64,2}}) at /julia/packages/Zygote/bdE6T/src/lib/lib.jl:132
 [13] #274#back at /julia/packages/ZygoteRules/Jn4LO/src/adjoint.jl:48 [inlined]
 [14] hvcat at ./abstractarray.jl:1662 [inlined]
 [15] step2 at /Notebooks/backprop.jl:147 [inlined]
 [16] (::typeof(βˆ‚(Main.Backprop.step2)))(::Tuple{Array{Float64,2},Array{Float64,2},UpperTriangular{Float64,Array{Float64,2}}}) at /julia/packages/Zygote/bdE6T/src/compiler/interface2.jl:0
 [17] top-level scope at In[8]:1
7

In the end, Zygote could not handle the code and I had to write my own adjoint. Maybe I will take a look at Zygote in the future.