# Convex Neural Network Using Skip Layers in Flux.jl
**URL:** https://discourse.julialang.org/t/convex-neural-network-using-skip-layers-in-flux-jl/87827
**Category:** Machine Learning
**Created:** [September 26, 2022, 5:31pm UTC](https://discourse.julialang.org/t/convex-neural-network-using-skip-layers-in-flux-jl/87827 "2022-09-26T17:31:46Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@marcofrancis](https://discourse.julialang.org/u/marcofrancis)
#### Post date: [September 26, 2022, 5:31pm UTC](https://discourse.julialang.org/t/convex-neural-network-using-skip-layers-in-flux-jl/87827/1 "2022-09-26T17:31:46Z")
Hi,
I’m trying to code up the following net, where y is the input of my net, f(y) is the output of the net, W^z\_k, W^y\_k are matrices and b\_k are vectors :
- f(y) = z\_k
- z\_k = softplus(W^z\_k z\_{k-1}+W^y\_k y+b\_k)
- …
- z\_{1} = softplus(W^y\_{0} y+b\_{0})
As you can see it’s basically a fully connected net, where each layer has access to the input. Furthermore, W^z\_k need to be positive (elementwise) and I was thinking of enforcing this applying say an exponential basically defining W^z\_k = \exp(W\_k), where now W\_k is an unconstrained matrix.
Somehow though I cannot find a way so that each layer in a chain keeps access to the input of the chain. Any idea on how to code this chain in Flux? I’m sure I can define a single layer that does the whole thing but it won’t be very practical.
---
### Author:  [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)
#### Post date: [September 26, 2022, 6:31pm UTC](https://discourse.julialang.org/t/convex-neural-network-using-skip-layers-in-flux-jl/87827/2 "2022-09-26T18:31:20Z")
If you want to use built-in layers for this see `SkipConnection` and `Parallel`: [Model Reference · Flux](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.SkipConnection)
---
### Author:  [@marcofrancis](https://discourse.julialang.org/u/marcofrancis)
#### Post date: [September 26, 2022, 6:59pm UTC](https://discourse.julialang.org/t/convex-neural-network-using-skip-layers-in-flux-jl/87827/3 "2022-09-26T18:59:22Z")
I’ve seen these, but how would you use them to create the architecture I mentioned? It’s not clear to me how to do it since after one skip layer the original input is lost
---
### Author:  [@mcabbott](https://discourse.julialang.org/u/mcabbott)
#### Post date: [September 26, 2022, 7:55pm UTC](https://discourse.julialang.org/t/convex-neural-network-using-skip-layers-in-flux-jl/87827/4 "2022-09-26T19:55:34Z")
I think this does what you write:
```julia
struct Adder{T<:Tuple}; layers::T; end;
Adder(layers...) = Adder(layers)
Flux.@functor Adder
function (a::Adder)(y)
d1 = a.layers[1]
z = d1(y)
for d in a.layers[2:end]
z = d(z + y)
end
z
end
m = Adder(Dense(2=>2, softplus), Dense(2=>2, softplus))
m(rand(2))
```
---
### Author:  [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)
#### Post date: [September 27, 2022, 12:32am UTC](https://discourse.julialang.org/t/convex-neural-network-using-skip-layers-in-flux-jl/87827/5 "2022-09-27T00:32:54Z")
```julia
demo_layer1(k) = y -> begin
@info "just y" k y
y
end
demo_layer2(k) = (z, y) -> begin
@info "y & z" k y z
k * (y + z)
end
# Easier to see the structure if we build up the model iteratively:
model = demo_layer1(1) # z1
model = SkipConnection(model, demo_layer2(2)) # feeding into z2
model = SkipConnection(model, demo_layer2(3)) # etc...
julia> model(1) # y = 1
┌ Info: just y
│ k = 1
└ y = 1
┌ Info: y & z
│ k = 2
│ y = 1
└ z = 1
┌ Info: y & z
│ k = 3
│ y = 1
└ z = 4
15
```
Note how the calculations for z\_{2+} are being done in the skip connection’s “connection” and not as the (non-identity) branch, while z\_1 _is_ being done as part of that branch. There are also ways of formulating the forward pass of your layer so you only need one for both equations, but that’s an aside.
---
### Author:  [@marcofrancis](https://discourse.julialang.org/u/marcofrancis)
#### Post date: [September 27, 2022, 8:30am UTC](https://discourse.julialang.org/t/convex-neural-network-using-skip-layers-in-flux-jl/87827/6 "2022-09-27T08:30:00Z")
Thanks, I’ll give a try to both solutions 🙂