# Separable convolution

**URL:** <https://discourse.julialang.org/t/separable-convolution/24141>\
**Category:** Machine Learning\
**Created:** [May 12, 2019, 3:19pm UTC](https://discourse.julialang.org/t/separable-convolution/24141 "2019-05-12T15:19:41Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [May 12, 2019, 3:19pm UTC](https://discourse.julialang.org/t/separable-convolution/24141/1 "2019-05-12T15:19:41Z")

</div>

Hi All,

I would like to run a separate over individual filters. Specifically imagine I have two matrices and I want to convolve each of them with its own filter. And I wonder if it possible to write it as a single convolution.  
MWE should be as follows

```julia
x,y = randn(3,3,1,1),randn(3,3,1,1)
Cx, Cy = Conv((2,2), 1 => 1), Conv((2,2), 1 => 1)

xy = cat(x, y, dims = 3)
C = ???
C(xy) = cat(Cx(x), Cy(y), dims = 3)

```

where `C = ???` stands for an operator I do not know how to write.

Thanks for an help in advance.  
Tomas

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [May 12, 2019, 3:57pm UTC](https://discourse.julialang.org/t/separable-convolution/24141/2 "2019-05-12T15:57:00Z")

</div>

I am not certain what all your notation means. But ImageFiltering.jl has support for separable convolution; you can define the kernel as a tuple of lower-dimensional objects that get applied sequentially. It even does pipeline optimization to improve cache utilization.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [May 12, 2019, 5:27pm UTC](https://discourse.julialang.org/t/separable-convolution/24141/3 "2019-05-12T17:27:12Z")

</div>

Sweet, I guess this is exactly what I am interested in. Of course, I would need compatibility with Flux.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [May 13, 2019, 2:41am UTC](https://discourse.julialang.org/t/separable-convolution/24141/4 "2019-05-13T02:41:33Z")

</div>

I think what you’re asking for is called a “depthwise convolution” in Flux:

```julia
C = DepthwiseConv((2,2), 2=>1)

```

The `2=>1` is really confusing here, but seems to map a two channel input to a two channel output with the property that changing `x` only changes `output[:,:,1,:]`, and changing `y` only changes `output[:,:,2,:]`.

Using `ImageFiltering` would probably work on the CPU if wrapped in an appropriate layer with the flux params plumbed through. But I suspect it would be a lot of work to get GPU training working.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [May 13, 2019, 2:51am UTC](https://discourse.julialang.org/t/separable-convolution/24141/5 "2019-05-13T02:51:31Z")

</div>

In figuring out the above, I ended up in a rabbit hole of neural network jargon and discovered some other things which might be useful to someone in the future. It’s a bit off topic, but having written most of it already I thought I’d post it here in case someone finds it useful.

It seems that the depthwise conv layer is a special case of what’s known as [grouped convolutions](https://medium.com/@yu4u/why-mobilenet-and-its-variants-e-g-shufflenet-are-fast-1c7048b9618d), where the number of groups equals the number of channels. I gather this terminology has been around for a while but was only just added to tensorflow recently (see [Add support for cudnn's group convolution. by ppwwyyxx · Pull Request #25818 · tensorflow/tensorflow · GitHub](https://github.com/tensorflow/tensorflow/pull/25818)).

AFAICT from looking at the Flux source this is not yet present in Flux (though I remain confused about the channel mapping behavior of DepthwiseConv). The following post from last year mentions grouped convolutions, and I see no sign that it’s been added in the meantime:

> **[ONNX.jl : The past, present and future.](https://medium.com/@ayush1999/onnx-jl-the-past-present-and-future-d3b497a0cd4c)**
>
> Machine learning is a more than a field, it’s a revolution. In the recent years, there has been a quantum leap in the number of companies…

For GPU training of grouped convolutions, I think something like the following would be required:

1. Add support for [`cudnnSetConvolutionGroupCount`](https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html#grouped-convolutions) to `CuArrays`.
2. Plumb this through NNlib.jl, presumably in analogy with what needed to be added for `DepthwiseConv` [Depthwise Convolutions by avik-pal · Pull Request #42 · FluxML/NNlib.jl · GitHub](https://github.com/FluxML/NNlib.jl/pull/42).
3. Add an API in Flux to support groups. This might be a `groups` keyword to `Conv` in analogy to similar high level frameworks.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [May 13, 2019, 5:41am UTC](https://discourse.julialang.org/t/separable-convolution/24141/6 "2019-05-13T05:41:17Z")

</div>

Thank you very much both for help. Intuitively, I was thinking that separable convolution is the thing I want, but frankly speaking, I was not sure. I am not expert in convolutions and I am not that much interested in CNNs. I just run into a small problem I am curious about.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [May 13, 2019, 5:56am UTC](https://discourse.julialang.org/t/separable-convolution/24141/7 "2019-05-13T05:56:23Z")

</div>

The term separable convolution is a well-established term in image processing and means that an N-dimensional kernel is replaced by a sequence of N 1-dimensional kernels, with the equivalent result. This is not possible for kernels in general but does work for a number of practically important ones, not least Gaussians and derivatives of Gaussians.

From what I can tell this is not what you are looking for.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [May 13, 2019, 6:15am UTC](https://discourse.julialang.org/t/separable-convolution/24141/8 "2019-05-13T06:15:27Z")

</div>

Yes, it looks like a pair of unrelated convolutions for each channel is desired. That’s definitely not a separable convolution.

One super confusing thing I found reading the NN jargon was that “depthwise convolution” seems to be different from “depthwise separable convolution”, with the latter being a “depthwise convolution” followed by a “1x1 convolution” (ie, “color mixing”; weights connecting all input channels to all output channels, being the same for every pixel).