# \[blog post\] Implement your own AD with Julia in ONE day

**URL:** <https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666>\
**Category:** Community\
**Tags:** blog-post\
**Created:** [October 23, 2018, 4:21am UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666 "2018-10-23T04:21:07Z")\
**Posts on this page:** 14\
**Page:** 2

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [October 23, 2018, 5:55pm UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/21 "2018-10-23T17:55:08Z")

</div>

Yes, if you just write

```julia
Zygote.@grad Base.:(*)(lhs::Matrix, rhs::Matrix) = gemm('N', 'N', lhs, rhs), grad -> (grad * transpose(rhs), transpose(lhs) * grad)

```

You will get an `gemm` not defined error, which means this is called.

This is probably not the gradient expression issue, but might be a Zygote bug somewhere.

---

<div class="post-metadata">

**Author:** ![improbable22](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/improbable22/32/5464_2.png) [@improbable22](https://discourse.julialang.org/u/improbable22)\
**Post date:** [October 23, 2018, 6:03pm UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/22 "2018-10-23T18:03:46Z")

</div>

OK, so much for that theory! Zygote is clearly above my pay grade…

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 23, 2018, 11:04pm UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/23 "2018-10-23T23:04:32Z")

</div>

> [@Roger-luo](#):
>
> If you are really a “practitioner”, under this situation, I believe you will choose Julia

I am really about a corporate practitioner who’s will just make use of whatever is already there and they don’t have the ability to come up with anything new. E.g. I know “senior data scientist” that don’t know how to fit a logistic regression, so I think you think to highly of these practitioners I am thinking about. But that’s an extreme case, a practitioner I am thinking about is someone like me who can make use of a LSTM, ResBlock, but wouldn’t at any point try to invent something new, I just know how to throw a few layers together and I roughly know what each layer is meant to do kinda practitioner.

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [October 23, 2018, 11:14pm UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/24 "2018-10-23T23:14:00Z")

</div>

Well, maybe [AutoML](https://cloud.google.com/automl/)? I don’t think throw a few layers together can make a paper, but people do need that in product. And that is actually why we are still supporting Python, and things like PyTorch is porting caffe2 for product, tensorflow is supporting training on mobiles, and there is even new framework written in C/C++ for ARMs etc.

I personally think, if someone are using Python, without Numba, C/C++, Cython, etc. then Python is fine and probably you won’t have much motivation in switching the language unless you want to use fresher techs.

And it is always fine to write in you favored language and framework, there is ONNX for model and many other formats for parameters.

And since it is way easier to implement new things in Julia, I believe at some point, there will be plenty of existing models here and then engineers without ML background will be able to use them in product. And that’s when everyone moves to Julia.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 24, 2018, 5:53am UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/25 "2018-10-24T05:53:48Z")

</div>

> [@xiaodai](#):
>
> “senior data scientist” that don’t know how to fit a logistic regression

I am not sure that catering to the demands of someone like this is a priority for many Julia packages at the moment. I don’t see a problem with this; a lot of commercial software exists precisely to fill in this niche.

---

<div class="post-metadata">

**Author:** ![improbable22](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/improbable22/32/5464_2.png) [@improbable22](https://discourse.julialang.org/u/improbable22)\
**Post date:** [October 24, 2018, 1:28pm UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/26 "2018-10-24T13:28:19Z")

</div>

One further comment on this benchmark: if the `tr` gradient returns `I` rather than a full matrix, then Zygote is much quicker.

```julia
using Zygote, Flux, LinearAlgebra, BenchmarkTools

f(x) = tr(x * x')
r = rand(30,30);
@btime Tracker.gradient(f,$r)[1][1:3] # 19.662 μs (128 allocations: 70.44 KiB)

Zygote.@grad LinearAlgebra.tr(x) = tr(x), Δ-> (Δ * I ,)
@btime Zygote.gradient(f,$r)[1][1:3] # 4.239 μs (7 allocations: 28.89 KiB)

Zygote.@grad LinearAlgebra.tr(x) = tr(x), Δ-> (Δ * Matrix(I, size(x)), )
@btime Zygote.gradient(f,$r)[1][1:3] # 37.053 μs (12 allocations: 37.27 KiB)

```

---

<div class="post-metadata">

**Author:** ![DoktorMike](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/doktormike/32/2736_2.png) [@DoktorMike](https://discourse.julialang.org/u/DoktorMike)\
**Post date:** [October 24, 2018, 8:07pm UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/27 "2018-10-24T20:07:27Z")

</div>

Impressive and elegant blog post. Thanks for writing it! 🙂  
A small note: you have a minor typo in your equation 3. I believe you meant for the 2 to be subscripted. It’s in the Compute-graph section. Thus changing y1=z2x to y\_1=z\_2 x. Similarly you have an issue in y2=b\cdot x which I believe should be y\_2=b\cdot x.

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [October 24, 2018, 8:41pm UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/28 "2018-10-24T20:41:33Z")

</div>

Well, I does not contain the shape info, which can cause error. And to be fair, AutoGrad.jl use `grad * Matrix{T}(I, size(output))` as well.

I think this is not the main problem. I’m still trying to understand how Zygote’s slow down happens, which was not what I expected.

---

<div class="post-metadata">

**Author:** ![DoktorMike](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/doktormike/32/2736_2.png) [@DoktorMike](https://discourse.julialang.org/u/DoktorMike)\
**Post date:** [November 3, 2018, 2:59pm UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/29 "2018-11-03T14:59:46Z")

</div>

Hey again,

Just playing around a bit with your nice AD package 🙂 . I’m trying a simple affine transformation for a small MLP. The following code leads to an error. I cannot really understand fully what’s going wrong but my guess would be that it doesn’t have a registered method for dealing with a matrix vector multiplication. Am I right about that or is there something else going on here?

```julia
using YAAD

function yaadderivtest(nhidden=100_000, ninput=1_000)
    W = Variable(rand(nhidden, ninput) .- 0.5)
    b = Variable(rand(nhidden) .- 0.5)
    x = rand(ninput)
    f(W, b) = sum(W*x .+ b)
    y = tr(f(W, b))
    backward(y)
end

yaadderivtest()

```

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [November 3, 2018, 3:42pm UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/30 "2018-11-03T15:42:20Z")

</div>

I think I implemented matrix multiplication, but sum is not implemented. Maybe you want to show me the error msg in the issue.

Best

Roger

---

<div class="post-metadata">

**Author:** ![DoktorMike](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/doktormike/32/2736_2.png) [@DoktorMike](https://discourse.julialang.org/u/DoktorMike)\
**Post date:** [November 3, 2018, 3:45pm UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/31 "2018-11-03T15:45:48Z")

</div>

> [@DoktorMike](#):
>
> function yaadderivtest(nhidden=100\_000, ninput=1\_000) W = Variable(rand(nhidden, ninput) .- 0.5) b = Variable(rand(nhidden) .- 0.5) x = rand(ninput) f(W, b) = sum(W\*x .+ b) y = tr(f(W, b)) backward(y) end yaadderivtest()

Sure, the error is this:

```julia
ERROR: MethodError: no method matching Array{T,2} where T(::LinearAlgebra.UniformScaling{Bool}, ::Tuple{})
Closest candidates are:
  Array{T,2} where T(::LinearAlgebra.UniformScaling, ::Integer, ::Integer) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/LinearAlgebra/src/uniformscaling.jl:325
  Array{T,2} where T(::LinearAlgebra.UniformScaling, ::Tuple{Int64,Int64}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/LinearAlgebra/src/uniformscaling.jl:326
Stacktrace:
 [1] gradient(::typeof(tr), ::Float64, ::Float64, ::Float64) at /home/michael/.julia/packages/YAAD/xLJz2/src/operators/linalg.jl:4
 [2] #gradient#3(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::YAAD.Trait.Method{typeof(tr)}, ::Float64, ::Float64, ::Float64) at /home/michael/.julia/packages/YAAD/xLJz2/src/comput_graph.jl:297
 [3] gradient(::YAAD.Trait.Method{typeof(tr)}, ::Float64, ::Float64, ::Float64) at /home/michael/.julia/packages/YAAD/xLJz2/src/comput_graph.jl:297
 [4] gradient(::CachedNode{Node{YAAD.Trait.Method{typeof(tr)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(sum)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(Base.Broadcast.materialize)},Tuple{CachedNode{Node{YAAD.Trait.Broadcasted{typeof(+)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(*)},Tuple{Variable{Array{Float64,2}},Array{Float64,1}},NamedTuple{(),Tuple{}}},Array{Float64,1}},Variable{Array{Float64,1}}},NamedTuple{(),Tuple{}}},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(+),Tuple{Array{Float64,1},Array{Float64,1}}}}},NamedTuple{(),Tuple{}}},Array{Float64,1}}},NamedTuple{(),Tuple{}}},Float64}},NamedTuple{(),Tuple{}}},Float64}, ::Float64) at /home/michael/.julia/packages/YAAD/xLJz2/src/comput_graph.jl:290
 [5] backward(::CachedNode{Node{YAAD.Trait.Method{typeof(tr)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(sum)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(Base.Broadcast.materialize)},Tuple{CachedNode{Node{YAAD.Trait.Broadcasted{typeof(+)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(*)},Tuple{Variable{Array{Float64,2}},Array{Float64,1}},NamedTuple{(),Tuple{}}},Array{Float64,1}},Variable{Array{Float64,1}}},NamedTuple{(),Tuple{}}},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(+),Tuple{Array{Float64,1},Array{Float64,1}}}}},NamedTuple{(),Tuple{}}},Array{Float64,1}}},NamedTuple{(),Tuple{}}},Float64}},NamedTuple{(),Tuple{}}},Float64}, ::Function, ::Float64) at /home/michael/.julia/packages/YAAD/xLJz2/src/comput_graph.jl:247
 [6] backward(::CachedNode{Node{YAAD.Trait.Method{typeof(tr)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(sum)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(Base.Broadcast.materialize)},Tuple{CachedNode{Node{YAAD.Trait.Broadcasted{typeof(+)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(*)},Tuple{Variable{Array{Float64,2}},Array{Float64,1}},NamedTuple{(),Tuple{}}},Array{Float64,1}},Variable{Array{Float64,1}}},NamedTuple{(),Tuple{}}},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(+),Tuple{Array{Float64,1},Array{Float64,1}}}}},NamedTuple{(),Tuple{}}},Array{Float64,1}}},NamedTuple{(),Tuple{}}},Float64}},NamedTuple{(),Tuple{}}},Float64}, ::YAAD.Trait.Method{typeof(tr)}, ::Float64) at /home/michael/.julia/packages/YAAD/xLJz2/src/comput_graph.jl:240
 [7] backward(::CachedNode{Node{YAAD.Trait.Method{typeof(tr)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(sum)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(Base.Broadcast.materialize)},Tuple{CachedNode{Node{YAAD.Trait.Broadcasted{typeof(+)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(*)},Tuple{Variable{Array{Float64,2}},Array{Float64,1}},NamedTuple{(),Tuple{}}},Array{Float64,1}},Variable{Array{Float64,1}}},NamedTuple{(),Tuple{}}},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(+),Tuple{Array{Float64,1},Array{Float64,1}}}}},NamedTuple{(),Tuple{}}},Array{Float64,1}}},NamedTuple{(),Tuple{}}},Float64}},NamedTuple{(),Tuple{}}},Float64}, ::Float64) at /home/michael/.julia/packages/YAAD/xLJz2/src/comput_graph.jl:239
 [8] backward(::CachedNode{Node{YAAD.Trait.Method{typeof(tr)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(sum)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(Base.Broadcast.materialize)},Tuple{CachedNode{Node{YAAD.Trait.Broadcasted{typeof(+)},Tuple{CachedNode{Node{YAAD.Trait.Method{typeof(*)},Tuple{Variable{Array{Float64,2}},Array{Float64,1}},NamedTuple{(),Tuple{}}},Array{Float64,1}},Variable{Array{Float64,1}}},NamedTuple{(),Tuple{}}},Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(+),Tuple{Array{Float64,1},Array{Float64,1}}}}},NamedTuple{(),Tuple{}}},Array{Float64,1}}},NamedTuple{(),Tuple{}}},Float64}},NamedTuple{(),Tuple{}}},Float64}) at /home/michael/.julia/packages/YAAD/xLJz2/src/comput_graph.jl:219
 [9] yaadderivtest(::Int64, ::Int64) at ./REPL[2]:7
 [10] yaadderivtest() at ./REPL[2]:2
 [11] top-level scope at none:0

```

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [November 3, 2018, 4:04pm UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/32 "2018-11-03T16:04:38Z")

</div>

Yes, this is because sum is not implemented. (See the backtrace). I’m still exploring to see if there’s an elegant way to implement all iterators’ backward propagation. Which is a problem when this is not a tracked type in AbtracyArray (while Tracker has different tracked type).

But if you want to use sum, simply register this function:

```julia
sum(x::AbstractNode)=register(sum, x)

gradient(::typeof(sum), grad:: Number, output, x::AbstractArray)=(fill!(similar(x), grad), )

```

Actually, I’m learning Cassette.jl and Zygote.jl recently, I believe that’ll be the future of ad.

Bests

Roger

---

<div class="post-metadata">

**Author:** ![DoktorMike](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/doktormike/32/2736_2.png) [@DoktorMike](https://discourse.julialang.org/u/DoktorMike)\
**Post date:** [November 3, 2018, 4:08pm UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/33 "2018-11-03T16:08:03Z")

</div>

Thanks for the reply. Indeed I’m trying Zygote.jl as well. And have my hopes up for Capstan.jl. 🙂

---

<div class="post-metadata">

**Author:** ![Roger-luo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/roger-luo/32/3399_2.png) [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)\
**Post date:** [November 3, 2018, 4:26pm UTC](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666/34 "2018-11-03T16:26:06Z")

</div>

Yeah, I might try how simple can it be to implement your own source 2source AD with cassette next time. lol.

[Previous page](https://discourse.julialang.org/t/blog-post-implement-your-own-ad-with-julia-in-one-day/16666.md?page=1)
