# Creating Parametric ReLU in Flux
**URL:** https://discourse.julialang.org/t/creating-parametric-relu-in-flux/10987
**Category:** Machine Learning
**Created:** [May 18, 2018, 12:50pm UTC](https://discourse.julialang.org/t/creating-parametric-relu-in-flux/10987 "2018-05-18T12:50:40Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Azamat](https://discourse.julialang.org/u/Azamat)
#### Post date: [May 18, 2018, 12:50pm UTC](https://discourse.julialang.org/t/creating-parametric-relu-in-flux/10987/1 "2018-05-18T12:50:40Z")
I would like to create Parametric ReLU (PReLU), an activation function, that is described in [[1502.01852] Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://arxiv.org/abs/1502.01852)
I know I should use
`using Flux.Tracker`
but I am a bit lost. My major challenge here is that for each layer of the network I need the trainable parameter ‘a’ of PReLU to be shared across the activations in that layer. So if the network has say 10 layers, then only 10 scalar trainable parameters should be added as a result (one for each layer).
Any ideas?
---
### Author:  [@jonathanBieler](https://discourse.julialang.org/u/jonathanBieler)
#### Post date: [May 22, 2018, 1:06pm UTC](https://discourse.julialang.org/t/creating-parametric-relu-in-flux/10987/3 "2018-05-22T13:06:30Z")
I’m not super familiar with Flux but this seems to work, I just modified a bit Flux’s `Dense` layer:
```julia
using Flux.Tracker, NNlib, Flux
struct DensePRELU{S,T,K}
W::S
b::T
a::K
end
prelu(x,a) = x > 0 ? x : a*x
function DensePRELU(in::Integer, out::Integer;
initW = Flux.glorot_uniform, initb = zeros)
return DensePRELU(param(initW(out, in)), param(initb(out)), param(0.0))
end
Flux.treelike(DensePRELU)
function (a::DensePRELU)(x)
W, b, a = a.W, a.b, a.a
NNlib.@fix prelu.(W*x .+ b, a)
end
m = Chain(
DensePRELU(10, 2),
)
M = rand(2,10)
fake_data() = begin x=rand(10); y = M*x; (x,y) end
train = [fake_data() for i=1:100]
loss(x, y) = sum(abs2.(m(x) .- y))
opt = ADAM(params(m))
evalcb = () -> println( mean( loss(d...) for d in train) )
for i=1:10 Flux.train!(loss, train, opt, cb=evalcb) end
```
Maybe there’s a way to directly do it with the default `Dense` layer.
---
### Author:  [@Azamat](https://discourse.julialang.org/u/Azamat)
#### Post date: [May 28, 2018, 5:33am UTC](https://discourse.julialang.org/t/creating-parametric-relu-in-flux/10987/4 "2018-05-28T05:33:51Z")
@jonathanBieler, Did you try running this? This produces the following error in the current stable version of Flux (v0.5.1):
```nohighlight
> opt = ADAM(params(m))
MethodError: Cannot `convert` an object of type Flux.Tracker.TrackedReal{Float64} to an object of type Flux.Optimise.Param
This may have arisen from a call to the constructor Flux.Optimise.Param(...),
since type constructors fall back to convert methods.
in ADAM at Flux/src/optimise/interface.jl:56
in optimiser at Flux/src/optimise/interface.jl:6
in collect at base/array.jl:476
in collect_to! at base/array.jl:518
in collect_to! at base/array.jl:508
```
---
### Author:  [@jonathanBieler](https://discourse.julialang.org/u/jonathanBieler)
#### Post date: [May 28, 2018, 8:03am UTC](https://discourse.julialang.org/t/creating-parametric-relu-in-flux/10987/5 "2018-05-28T08:03:10Z")
It works on v0.4.1 yes, I’m not sure what’s going on with the new release but you can fix the error with this:
```julia
function DensePRELU(in::Integer, out::Integer;
initW = Flux.glorot_uniform, initb = zeros)
return DensePRELU(param(initW(out, in)), param(initb(out)), param(zeros(1)))
end
```
It defines `a` as a 1 element vector instead of a float.
---
### Author:  [@Azamat](https://discourse.julialang.org/u/Azamat)
#### Post date: [May 28, 2018, 9:07am UTC](https://discourse.julialang.org/t/creating-parametric-relu-in-flux/10987/6 "2018-05-28T09:07:36Z")
@MikeInnes, this behaviour in the version 0.5.1 seems to me as a bug. Or is it intended?
---
### Author:  [@Azamat](https://discourse.julialang.org/u/Azamat)
#### Post date: [May 28, 2018, 2:16pm UTC](https://discourse.julialang.org/t/creating-parametric-relu-in-flux/10987/7 "2018-05-28T14:16:56Z")
Thanks @jonathanBieler!
I’ve modified your solution below to abstract it away from a particular layer, so that now it can be called with `BatchNorm` for example.
```nohighlight
using Flux
using Flux: Tracker, treelike
using NNlib
struct PReLU{T}
a::T
end
PReLU(init::Real) = PReLU(param([init/1]))
PReLU() = PReLU(0.0)
treelike(PReLU)
prelu(x, a) = x > 0 ? x : a*x
function (f::PReLU)(x)
NNlib.@fix prelu.(x, f.a)
end
m = Chain( Dense(10, 2), PReLU())
M = rand(2,10)
fake_data() = begin x=rand(10); y = M*x; (x,y) end
train = [fake_data() for i=1:100]
loss(x, y) = sum(abs2.(m(x) .- y))
opt = ADAM(params(m))
evalcb = () -> println( mean( loss(d...) for d in train) )
@time for i=1:10 Flux.train!(loss, train, opt, cb=evalcb) end
```
---
### Author:  [@jonathanBieler](https://discourse.julialang.org/u/jonathanBieler)
#### Post date: [May 28, 2018, 3:22pm UTC](https://discourse.julialang.org/t/creating-parametric-relu-in-flux/10987/8 "2018-05-28T15:22:12Z")
That’s better yeah. I you find that this parametric relu is helping, maybe do a PR to Flux to add it.
---
### Author:  [@cirobr](https://discourse.julialang.org/u/cirobr)
#### Post date: [February 20, 2024, 7:14pm UTC](https://discourse.julialang.org/t/creating-parametric-relu-in-flux/10987/9 "2024-02-20T19:14:28Z")
Cheers, hope I can get back to this old topic.
In current Flux v0.14 there is the `Scale` layer. I wonder if `prelu` could be implemented like this?
> prelu=Scale(1, relu, bias=false)
Thanks.
---
### Author:  [@Yang-yang](https://discourse.julialang.org/u/Yang-yang)
#### Post date: [February 1, 2025, 6:19pm UTC](https://discourse.julialang.org/t/creating-parametric-relu-in-flux/10987/10 "2025-02-01T18:19:48Z")
The parametric activation function can be realized in Lux.jl
```julia
model=@compact(w1=Dense(10, 2), a=[0.01f0]) do x
out = leakyrelu.(w1(x),a)
@return out
end
```
The output gives
```julia
@compact(
w1 = Dense(10 => 2), # 22 parameters
a = 1-element Vector{Float32},
) do x
out = leakyrelu.(w1(x), a)
return out
end # Total: 23 parameters,
# plus 0 states.
```