# \`Conv\` is 2x slow than pytorch \`Conv\` on cpu

**URL:** <https://discourse.julialang.org/t/conv-is-2x-slow-than-pytorch-conv-on-cpu/39802>\
**Category:** Machine Learning\
**Created:** [May 20, 2020, 5:46am UTC](https://discourse.julialang.org/t/conv-is-2x-slow-than-pytorch-conv-on-cpu/39802 "2020-05-20T05:46:41Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![yiyuezhuo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yiyuezhuo/32/27101_2.png) [@yiyuezhuo](https://discourse.julialang.org/u/yiyuezhuo)\
**Post date:** [May 20, 2020, 5:46am UTC](https://discourse.julialang.org/t/conv-is-2x-slow-than-pytorch-conv-on-cpu/39802/1 "2020-05-20T05:46:41Z")

</div>

Example:

```julia
using BenchmarkTools

using Flux
conv = Conv((7,7), 3 => 64; stride=2, pad=3)
dummy = randn(Float32, 224, 224, 3, 2);
conv(dummy) # ignore compile time
@benchmark conv(dummy) 
#=
BenchmarkTools.Trial: 
  memory estimate: 19.29 MiB
  allocs estimate: 60
  --------------
  minimum time: 22.996 ms (0.00% GC)
  median time: 25.009 ms (0.00% GC)
  mean time: 30.242 ms (17.79% GC)
  maximum time: 66.116 ms (62.37% GC)
  --------------
  samples: 165
  evals/sample: 1
=#

using PyCall
const torch = pyimport_conda("torch", "torch")
const nn = pyimport("torch.nn")
dummy_py = torch.randn(2, 3, 224, 224);
conv1_py = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=true)
conv1_py(dummy_py)
@benchmark conv1_py(dummy_py)  

#=
BenchmarkTools.Trial: 
  memory estimate: 160 bytes
  allocs estimate: 4
  --------------
  minimum time: 8.824 ms (0.00% GC)
  median time: 9.436 ms (0.00% GC)
  mean time: 10.463 ms (0.00% GC)
  maximum time: 19.719 ms (0.00% GC)
  --------------
  samples: 478
  evals/sample: 1
=#

```

Is the result expected, or a benchmark/implementation mistake? This is the first layer of ResNet18 (with extra bias), the full network implementation gets similar results.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [May 20, 2020, 8:54am UTC](https://discourse.julialang.org/t/conv-is-2x-slow-than-pytorch-conv-on-cpu/39802/2 "2020-05-20T08:54:45Z")

</div>

Something is fishy with that benchmark, how can pytorch not allocate more than 160 bytes when the output array is much larger than that?

---

<div class="post-metadata">

**Author:** ![yiyuezhuo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yiyuezhuo/32/27101_2.png) [@yiyuezhuo](https://discourse.julialang.org/u/yiyuezhuo)\
**Post date:** [May 20, 2020, 9:16am UTC](https://discourse.julialang.org/t/conv-is-2x-slow-than-pytorch-conv-on-cpu/39802/3 "2020-05-20T09:16:08Z")

</div>

😅 Sadly I didn’t even detect it before posting. But just `@time` also gives similar results, so I guess that the 2x gap itself could exist.

---

<div class="post-metadata">

**Author:** ![LeoKoo](https://avatars.discourse-cdn.com/v4/letter/l/ccd318/32.png) [@LeoKoo](https://discourse.julialang.org/u/LeoKoo)\
**Post date:** [May 21, 2020, 10:37am UTC](https://discourse.julialang.org/t/conv-is-2x-slow-than-pytorch-conv-on-cpu/39802/4 "2020-05-21T10:37:11Z")

</div>

This is probably not responsible for the entire 2x speed difference, but I have noticed Flux.Conv layers are not type-stable, which I guess would reduce performance. Opened a github issue [here](https://github.com/FluxML/Flux.jl/issues/1178) and a thread [here](https://discourse.julialang.org/t/flux-convolutional-layer-not-type-stable/39454).

Because of the lack of response I am unsure whether this type-instability is actually a big deal though, I am quite new to Julia.

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [May 21, 2020, 11:40am UTC](https://discourse.julialang.org/t/conv-is-2x-slow-than-pytorch-conv-on-cpu/39802/5 "2020-05-21T11:40:02Z")

</div>

Could one reason be that PyTorch uses the [CUDNN function which also adds bias and activation](https://docs.nvidia.com/deeplearning/sdk/cudnn-api/index.html#cudnnConvolutionBiasActivationForward) while Flux only does the [convolution with CUDNN](https://docs.nvidia.com/deeplearning/sdk/cudnn-api/index.html#cudnnConvolutionForward) and relies on generic implementations for adding bias performing activation?

---

<div class="post-metadata">

**Author:** ![LeoKoo](https://avatars.discourse-cdn.com/v4/letter/l/ccd318/32.png) [@LeoKoo](https://discourse.julialang.org/u/LeoKoo)\
**Post date:** [May 21, 2020, 12:17pm UTC](https://discourse.julialang.org/t/conv-is-2x-slow-than-pytorch-conv-on-cpu/39802/6 "2020-05-21T12:17:16Z")

</div>

Bias calculation could make a difference, but the benchmark uses the identity function activation so I think it would be compiled away and have zero overhead, no?

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [May 21, 2020, 12:37pm UTC](https://discourse.julialang.org/t/conv-is-2x-slow-than-pytorch-conv-on-cpu/39802/7 "2020-05-21T12:37:59Z")

</div>

Hmm, I just saw that the benchmark does not even use the GPU, so there is no CUDNN involved. Sorry for the noise.

Could it be an MKL vs OpenBLAS thing then?

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [September 5, 2020, 9:28pm UTC](https://discourse.julialang.org/t/conv-is-2x-slow-than-pytorch-conv-on-cpu/39802/8 "2020-09-05T21:28:46Z")

</div>

Do you mind trying again with the functional conv functions in NNlib/PyTorch? I think it would be good to open an issue in Flux or NNlib, so this would help narrow things down.

**Edit** : using MKL.jl helps narrow the gap on my machine:

```python
In [1]: import torch.nn.functional as F

In [2]: import torch

In [3]: x = torch.randn(2, 3, 224, 224)

In [4]: conv1 = torch.nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=True)

In [5]: %timeit conv1(x)
2.78 ms ± 70.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```

```julia
julia> using LinearAlgebra: BLAS

julia> BLAS.vendor()
:mkl

julia> using BenchmarkTools, Flux

julia> const x = randn(Float32, 224, 224, 3, 2);

julia> const conv1 = Conv((7,7), 3 => 64; stride=2, pad=3);

julia> @benchmark conv1($x)
BenchmarkTools.Trial: 
  memory estimate: 19.29 MiB
  allocs estimate: 45
  --------------
  minimum time: 4.588 ms (0.00% GC)
  median time: 5.330 ms (0.00% GC)
  mean time: 5.424 ms (4.99% GC)
  maximum time: 10.235 ms (7.86% GC)
  --------------
  samples: 922
  evals/sample: 1

```

---

<div class="post-metadata">

**Author:** ![yiyuezhuo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yiyuezhuo/32/27101_2.png) [@yiyuezhuo](https://discourse.julialang.org/u/yiyuezhuo)\
**Post date:** [September 6, 2020, 2:44am UTC](https://discourse.julialang.org/t/conv-is-2x-slow-than-pytorch-conv-on-cpu/39802/9 "2020-09-06T02:44:30Z")

</div>

Have opened an issue here:  
[https://github.com/FluxML/NNlib.jl/issues/234](https://github.com/FluxML/NNlib.jl/issues/234)

---

<div class="post-metadata">

**Author:** ![RoyiAvital](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/royiavital/32/571_2.png) [@RoyiAvital](https://discourse.julialang.org/u/RoyiAvital)\
**Post date:** [September 6, 2020, 5:43am UTC](https://discourse.julialang.org/t/conv-is-2x-slow-than-pytorch-conv-on-cpu/39802/10 "2020-09-06T05:43:34Z")

</div>

Wouldn’t PyTorch use what used to be MKL DNN (Now [OneDNN](https://github.com/oneapi-src/oneDNN))?