# Implementation of Spectral Normalization for Machine Learning

**URL:** <https://discourse.julialang.org/t/implementation-of-spectral-normalization-for-machine-learning/76074>\
**Category:** General Usage\
**Created:** [February 9, 2022, 12:39pm UTC](https://discourse.julialang.org/t/implementation-of-spectral-normalization-for-machine-learning/76074 "2022-02-09T12:39:09Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hareruya](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hareruya/32/47536_2.png) [@Hareruya](https://discourse.julialang.org/u/Hareruya)\
**Post date:** [February 9, 2022, 12:39pm UTC](https://discourse.julialang.org/t/implementation-of-spectral-normalization-for-machine-learning/76074/1 "2022-02-09T12:39:09Z")

</div>

Greetings.

I have migrated most of my Machine Learning work from working in Pytorch for Flux and have not felt the need to look back(especially as I move towards SciML), however there was a small hiccup I’d like t inquire about:

Spectral normalization (SN) is a technique employed in Machine Learning, specifically Generative Adversarial Networks, to stabilize learning of the discriminator. As an exercise, I tried to generate my own Dense layer function with spectral normalization. However, I still find myself unable to train the network because my implementation of the spectral norm seems to not be differentiable using the Flux rendition of the gradient function, but shows no problem with the ForwardDiff gradient function.

Below is a simple code I wrote to illustrate this fact:

```julia
using Flux: gradient
using GenericLinearAlgebra: svdvals

function snorm(X)
    return svdvals(X)[1]
end

X = rand(3,5)#Generating a random matrix to analyze.

println(snorm(X))#This part executes without issue.

println(gradient(snorm, X))#This part gives a "Can't differentiate foreigncall expression" error.

```

The stacktrace is as follows:

```julia
ERROR: Can't differentiate foreigncall expression
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] Pullback
    @ /usr/share/julia/stdlib/v1.7/LinearAlgebra/src/lapack.jl:1667 [inlined]
  [3] (::typeof(∂(gesdd!)))(Δ::Tuple{Nothing, Zygote.OneElement{Float64, 1, Tuple{Int64}, Tuple{Base.OneTo{Int64}}}, Nothing})
    @ Zygote ~/.julia/packages/Zygote/FPUm3/src/compiler/interface2.jl:0
  [4] Pullback
    @ /usr/share/julia/stdlib/v1.7/LinearAlgebra/src/svd.jl:211 [inlined]
  [5] (::typeof(∂(svdvals!)))(Δ::Zygote.OneElement{Float64, 1, Tuple{Int64}, Tuple{Base.OneTo{Int64}}})
    @ Zygote ~/.julia/packages/Zygote/FPUm3/src/compiler/interface2.jl:0
  [6] Pullback
    @ /usr/share/julia/stdlib/v1.7/LinearAlgebra/src/svd.jl:238 [inlined]
  [7] (::typeof(∂(svdvals)))(Δ::Zygote.OneElement{Float64, 1, Tuple{Int64}, Tuple{Base.OneTo{Int64}}})
    @ Zygote ~/.julia/packages/Zygote/FPUm3/src/compiler/interface2.jl:0
  [8] Pullback
    @ ./REPL[10]:2 [inlined]
  [9] (::Zygote.var"#57#58"{typeof(∂(snorm))})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/FPUm3/src/compiler/interface.jl:41
 [10] gradient(f::Function, args::Matrix{Float64})
    @ Zygote ~/.julia/packages/Zygote/FPUm3/src/compiler/interface.jl:76
 [11] top-level scope
    @ REPL[13]:1
 [12] top-level scope
    @ ~/.julia/packages/CUDA/bki2w/src/initialization.jl:52

```

However, if the exact same steps are taken using the gradient from ForwardDiff instead, there is no error reported.

```julia
using ForwardDiff: gradient
using GenericLinearAlgebra: svdvals

function snorm(X)
    return svdvals(X)[1]
end

X = rand(3,5)#Generating a random matrix to analyze.

println(snorm(X))#This part executes without issue.

println(gradient(snorm, X))#This part does not give a "Can't differentiate foreigncall expression" error anymore.

```

My research machine is an Arch Linux using the Linux-bin package downloaded from the AUR, and otherwise runs flawlessly. The error is reproduced in my personal machine, which is a Mac OS system. Both Julia distributions are up-to-date, and the packages were also updated yesterday.

Any help, be it correcting some error I might have made or suggesting an alternative implementation that would work, is appreciated. This is my first time writing anything here, so please do point out if I did anything out of line.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [February 9, 2022, 1:17pm UTC](https://discourse.julialang.org/t/implementation-of-spectral-normalization-for-machine-learning/76074/2 "2022-02-09T13:17:40Z")

</div>

The norm you’re using is called `opnorm` in Julia, perhaps this function is differentiable?

---

<div class="post-metadata">

**Author:** ![Hareruya](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hareruya/32/47536_2.png) [@Hareruya](https://discourse.julialang.org/u/Hareruya)\
**Post date:** [February 9, 2022, 1:30pm UTC](https://discourse.julialang.org/t/implementation-of-spectral-normalization-for-machine-learning/76074/3 "2022-02-09T13:30:55Z")

</div>

Thank you for the idea. I didn’t think to look up whether it was implemented as opnorm since the spectral norm is a specific type of operation norm.(Edit: Sorry, realized it works the same as the regular norm in that you inform the order. However, default seems to be 2, so everything here stands.)  
Unfortunately, the results seem to be the same in the end.

```julia
ERROR: Can't differentiate foreigncall expression
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] Pullback
    @ /usr/share/julia/stdlib/v1.7/LinearAlgebra/src/lapack.jl:1667 [inlined]
  [3] (::typeof(∂(gesdd!)))(Δ::Tuple{Nothing, Zygote.OneElement{Float64, 1, Tuple{Int64}, Tuple{Base.OneTo{Int64}}}, Nothing})
    @ Zygote ~/.julia/packages/Zygote/FPUm3/src/compiler/interface2.jl:0
  [4] Pullback
    @ /usr/share/julia/stdlib/v1.7/LinearAlgebra/src/svd.jl:211 [inlined]
  [5] (::typeof(∂(svdvals!)))(Δ::Zygote.OneElement{Float64, 1, Tuple{Int64}, Tuple{Base.OneTo{Int64}}})
    @ Zygote ~/.julia/packages/Zygote/FPUm3/src/compiler/interface2.jl:0
  [6] Pullback
    @ /usr/share/julia/stdlib/v1.7/LinearAlgebra/src/svd.jl:238 [inlined]
  [7] (::typeof(∂(svdvals)))(Δ::Zygote.OneElement{Float64, 1, Tuple{Int64}, Tuple{Base.OneTo{Int64}}})
    @ Zygote ~/.julia/packages/Zygote/FPUm3/src/compiler/interface2.jl:0
  [8] Pullback
    @ /usr/share/julia/stdlib/v1.7/LinearAlgebra/src/generic.jl:708 [inlined]
  [9] (::typeof(∂(opnorm2)))(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/FPUm3/src/compiler/interface2.jl:0
 [10] Pullback
    @ /usr/share/julia/stdlib/v1.7/LinearAlgebra/src/generic.jl:768 [inlined]
 [11] (::typeof(∂(opnorm)))(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/FPUm3/src/compiler/interface2.jl:0
 [12] Pullback
    @ /usr/share/julia/stdlib/v1.7/LinearAlgebra/src/generic.jl:767 [inlined]
 [13] (::Zygote.var"#57#58"{typeof(∂(opnorm))})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/FPUm3/src/compiler/interface.jl:41
 [14] gradient(f::Function, args::Matrix{Float64})
    @ Zygote ~/.julia/packages/Zygote/FPUm3/src/compiler/interface.jl:76
 [15] dopnorm(X::Matrix{Float64})
    @ Main ./REPL[4]:1
 [16] top-level scope
    @ REPL[5]:1
 [17] top-level scope
    @ ~/.julia/packages/CUDA/bki2w/src/initialization.jl:52

```

---

<div class="post-metadata">

**Author:** ![trahflow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/trahflow/32/30585_2.png) [@trahflow](https://discourse.julialang.org/u/trahflow)\
**Post date:** [February 9, 2022, 1:35pm UTC](https://discourse.julialang.org/t/implementation-of-spectral-normalization-for-machine-learning/76074/4 "2022-02-09T13:35:10Z")

</div>

The problem is that there is currently no `rrule` defined for `svdvals`.  
A workaround would be to define:

```julia
function snorm(X)
    return svd(X).S[1]
end

```

Which does a full svd (for which there is an `rrule` defined)

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [February 9, 2022, 1:36pm UTC](https://discourse.julialang.org/t/implementation-of-spectral-normalization-for-machine-learning/76074/5 "2022-02-09T13:36:35Z")

</div>

> [@baggepinnen](#):
>
> The norm you’re using is called `opnorm` in Julia, perhaps this function is differentiable

Interesting. This reminds me of [this thread](https://discourse.julialang.org/t/automatic-differentiation-of-complex-matrix-fails/75733) where I learned that at least differentiation of real `svd` seems supported by Zygote?

---

<div class="post-metadata">

**Author:** ![Hareruya](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hareruya/32/47536_2.png) [@Hareruya](https://discourse.julialang.org/u/Hareruya)\
**Post date:** [February 9, 2022, 1:39pm UTC](https://discourse.julialang.org/t/implementation-of-spectral-normalization-for-machine-learning/76074/6 "2022-02-09T13:39:02Z")

</div>

Thank you very much! It worked perfectly now!

---

<div class="post-metadata">

**Author:** ![mcabbott](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcabbott/32/6603_2.png) [@mcabbott](https://discourse.julialang.org/u/mcabbott)\
**Post date:** [February 9, 2022, 1:57pm UTC](https://discourse.julialang.org/t/implementation-of-spectral-normalization-for-machine-learning/76074/7 "2022-02-09T13:57:27Z")

</div>

The rule for SVD is [here](https://github.com/JuliaDiff/ChainRules.jl/blob/3590f9421950508a97d5a9dbc207208e331c8b75/src/rulesets/LinearAlgebra/factorization.jl#L221-L225). It would probably be very simple to add one for `svdvals`, perhaps calling the same function to do the work `svd_rev(nt, NoTangent(), S̄, NoTangent())` or perhaps just keeping the few lines which are nonzero there.

---

<div class="post-metadata">

**Author:** ![Hareruya](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hareruya/32/47536_2.png) [@Hareruya](https://discourse.julialang.org/u/Hareruya)\
**Post date:** [February 9, 2022, 2:15pm UTC](https://discourse.julialang.org/t/implementation-of-spectral-normalization-for-machine-learning/76074/8 "2022-02-09T14:15:44Z")

</div>

Thank you for the explanation and link.  
Using the same would probably be the way to go, since to my knowledge calculating the gradient of the spectral norm (where it exists) requires knowing the U and V matrices obtained from performing the standard svd.