# GPU performance problem of locally connected layer with Tullio

**URL:** <https://discourse.julialang.org/t/gpu-performance-problem-of-locally-connected-layer-with-tullio/85469>\
**Category:** Machine Learning\
**Tags:** tullio\
**Created:** [August 8, 2022, 11:36am UTC](https://discourse.julialang.org/t/gpu-performance-problem-of-locally-connected-layer-with-tullio/85469 "2022-08-08T11:36:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rasmus\_Hoier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rasmus_hoier/32/24036_2.png) [@Rasmus\_Hoier](https://discourse.julialang.org/u/Rasmus_Hoier)\
**Post date:** [August 8, 2022, 11:36am UTC](https://discourse.julialang.org/t/gpu-performance-problem-of-locally-connected-layer-with-tullio/85469/1 "2022-08-08T11:36:08Z")

</div>

I am using Tullio.jl to implement a locally connected layer, which is like a convolutional layer except there isn’t any weight sharing. I found Tullio very useful for this task, but unfortunately there is a large gap in performance between CPU and GPU. On CPU my convlocal function is nearly as fast as NNlib.conv. On GPU the forwards pass is fast, but the backwards pass is very slow. I could use tips on how to improve the performance of the gradient computation on GPU.

Here is a MWE:

```julia
using Tullio, NNlib, Flux, BenchmarkTools, CUDA, CUDAKernels, KernelAbstractions

function convlocal(x, W)
    @tullio fastmath=false c[s, t, c2, b] := x[s+i-1, t+j-1, c1, b] * W[s+i-1, t+j-1, i, j, c1, c2]
    return c
end

device = gpu
kernel_width, kernel_height, ch_in, ch_out = 3, 3, 16, 16
img_width, img_height = 28, 28
batchsize = 100

x = device(rand(Float32, img_width, img_height, ch_in, batchsize))
Wconv = device(rand(Float32, kernel_width, kernel_height, ch_in, ch_out))
Wlocal = device(rand(Float32, img_width, img_height, kernel_width, kernel_height, ch_in, ch_out))
ps_conv = Flux.params(Wconv)
ps_local = Flux.params(Wlocal)

@info "Benchmarking convlocal"
@info "forward"
@btime convlocal(x, Wlocal)
@info "backward"
@btime gradient(() -> sum(abs2, convlocal(x, Wlocal)), ps_local)
println("=====================")
@info "Benchmarking NNlib.conv"
@info "forward"
@btime conv(x, Wconv)
@info "backward"
@btime gradient(() -> sum(abs2, NNlib.conv(x, Wconv)), ps_conv)
println("=====================")

```

Which on my machine gives the output:

```julia
[ Info: Benchmarking convlocal
[ Info: forward
  24.326 μs (151 allocations: 7.19 KiB)
[ Info: backward
  3.036 s (629 allocations: 31.00 KiB)
=====================
[ Info: Benchmarking NNlib.conv
[ Info: forward
  17.402 μs (75 allocations: 2.75 KiB)
[ Info: backward
  325.009 μs (401 allocations: 23.81 KiB)
=====================

```

---

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [August 18, 2022, 9:08am UTC](https://discourse.julialang.org/t/gpu-performance-problem-of-locally-connected-layer-with-tullio/85469/2 "2022-08-18T09:08:31Z")

</div>

Have you considered `@ein`? In a brief test, I found that `@ein` was faster than `@tullio` on GPU with a 1×1 filter, please see [here](https://discourse.julialang.org/t/flux-locally-connected-layer-performance/55711). Would be great to a fast locally connected layer in Flux.

---

<div class="post-metadata">

**Author:** ![Rasmus\_Hoier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rasmus_hoier/32/24036_2.png) [@Rasmus\_Hoier](https://discourse.julialang.org/u/Rasmus_Hoier)\
**Post date:** [August 19, 2022, 11:31am UTC](https://discourse.julialang.org/t/gpu-performance-problem-of-locally-connected-layer-with-tullio/85469/3 "2022-08-19T11:31:32Z")

</div>

Thanks,  
I had a look at @ein (from OMEinsum right?), but I could not get it to work with 3x3 filters. Also it didn’t seem to work with autodiff, so I would need to define a custom rrule anyways.

So for now since Tullio was only slow on gpu in the backwards pass I am trying out wrapping the symbolic gradients (provided by @Tullio verbose=1) in their own functions. This solves the GPU runtime issue.

But when comparing the gradients produced by the explicitly defined functions to those produced by Tullios auto generated functions, there is some difference. Gradients with respect to W always match, but gradients with respect to x only match when ch\_out=1.

Here is a small cpu based MWE:

```julia
using Tullio, Zygote

function convlocal(x, W)
    @tullio verbose=0 fastmath=false c[s, t, c2, b] := x[s+i-1, t+j-1, c1, b] * W[s+i-1, t+j-1, i, j, c1, c2]
    return c
end

function ∇W(x, Δy, ΔW)
    @tullio verbose=0 fastmath=false grad=false (ΔW[(s + i) - 1, (t + j) - 1, i, j, c1, c2] = ΔW[(s + i) - 1, (t + j) - 1, i, j, c1, c2] + Δy[s, t, c2, b] * conj(x[(s + i) - 1, (t + j) - 1, c1, b]))
end

function ∇x(W, Δy, Δx)
    @tullio verbose=0 fastmath=false grad=false (Δx[(s + i) - 1, (t + j) - 1, c1, b] = Δx[(s + i) - 1, (t + j) - 1, c1, b] + Δy[s, t, c2, b] * conj(W[(s + i) - 1, (t + j) - 1, i, j, c1, c2]))
end

kernel_width, kernel_height, ch_in, ch_out = 3, 3, 2, 3
img_width, img_height = 10, 10
batchsize = 3
x = rand(Float32, img_width, img_height, ch_in, batchsize)
W = rand(Float32, img_width, img_height, kernel_width, kernel_height, ch_in, ch_out)
Δy = rand(Float32, img_width-2, img_height-2, ch_out, batchsize)
ΔW = zeros(Float32, size(W))   
Δx = zeros(Float32, size(x))
       
# Compute grads wrt x and W using the explicitly defined functions
∇x(W, Δy, Δx)
∇W(x, Δy, ΔW)

# compute grads wrt x and W using Tullios auto generated gradients
G = Zygote._pullback(convlocal, x, W)
Δx2 = G[2](Δy)[2]
ΔW2 = G[2](Δy)[3]

# compare the two approaches
@show Δx ≈ Δx2
@show ΔW ≈ ΔW2

```

Producing the output

```julia
Δx ≈ Δx2 = false
ΔW ≈ ΔW2 = true

```

---

<div class="post-metadata">

**Author:** ![Rasmus\_Hoier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rasmus_hoier/32/24036_2.png) [@Rasmus\_Hoier](https://discourse.julialang.org/u/Rasmus_Hoier)\
**Post date:** [August 23, 2022, 8:01am UTC](https://discourse.julialang.org/t/gpu-performance-problem-of-locally-connected-layer-with-tullio/85469/4 "2022-08-23T08:01:34Z")

</div>

I got some useful feedback from the Tullio dev regarding the issue with ∇x [here](https://github.com/mcabbott/Tullio.jl/issues/153).  
Also I discovered that timing on GPU requires using CUDA.@sync, so gpu benchmarks look a bit different, but still slow compared to NNlib.conv.

Long story short: Tullio does an extremely good job for locally connected layers on the CPU, but is quite slow on the gpu (both forwards and backwards). An efficient GPU implementation probably needs to be written directly either in KernelAbstractions.jl or CUDA.jl. However, the KernelAbstractions code produced by Tullio (printed when setting @tullio verbose=2) is a useful starting point.

Sidenote: An alternative approach is the fold/unfold based approach proposed in pytorch: [link1](https://discuss.pytorch.org/t/locally-connected-layers/26979), [link2](https://github.com/pytorch/pytorch/issues/499)(though this remains unreleased for several years now). The problem with this approach is that locally connected layers already require **much** more memory than the corresponding convolutional layer, and using unfold could make this memory issue worse. On top of that memory is often the bottleneck on GPU.