# Impose initialization adn normalization on layers in Flux

**URL:** <https://discourse.julialang.org/t/impose-initialization-adn-normalization-on-layers-in-flux/23921>\
**Category:** Machine Learning\
**Tags:** first-steps\
**Created:** [May 6, 2019, 4:19pm UTC](https://discourse.julialang.org/t/impose-initialization-adn-normalization-on-layers-in-flux/23921 "2019-05-06T16:19:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![stefesse](https://avatars.discourse-cdn.com/v4/letter/s/a88e57/32.png) [@stefesse](https://discourse.julialang.org/u/stefesse)\
**Post date:** [May 6, 2019, 4:19pm UTC](https://discourse.julialang.org/t/impose-initialization-adn-normalization-on-layers-in-flux/23921/1 "2019-05-06T16:19:01Z")

</div>

Hi all, I am constructing a simple perceptron with Flux, something like: y = W\*x + b. To do so I simply use:

```julia
model = Dense(N,1,nonlinearity);

```

What I do not understand is:

- How can I impose the initial value for the matrix W and b?
- How can impose that at every iteration in the learning norm(W) = 1?  
Thanks!

---

<div class="post-metadata">

**Author:** ![jbrea](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jbrea/32/3879_2.png) [@jbrea](https://discourse.julialang.org/u/jbrea)\
**Post date:** [May 7, 2019, 7:27am UTC](https://discourse.julialang.org/t/impose-initialization-adn-normalization-on-layers-in-flux/23921/2 "2019-05-07T07:27:20Z")

</div>

Hi

To set initial values you can do

```julia
W = rand(1, N); b = rand(1);
model = Dense(param(W), param(b), nonlinearity);

```

This is also roughly what happens when you call `model = Dense(N,1,nonlinearity);`, as you can see by looking at the source with

```julia
@edit model = Dense(N,1,nonlinearity);

```

One option to keep norm(W) = 1 would be to define a custom wrapper, e.g.

```julia
struct NormedDense{D}
  l::D
end
function (a::NormedDense)(x::AbstractArray)
  W, b, σ = a.l.W ./ norm(a.l.W), a.l.b, a.l.σ
  σ.(W*x .+ b)
end
model = NormedDense(Dense(param(W), param(b), nonlinearity));

```

---

<div class="post-metadata">

**Author:** ![williamfgc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/williamfgc/32/15445_2.png) [@williamfgc](https://discourse.julialang.org/u/williamfgc)\
**Post date:** [September 11, 2020, 8:48pm UTC](https://discourse.julialang.org/t/impose-initialization-adn-normalization-on-layers-in-flux/23921/3 "2020-09-11T20:48:35Z")

</div>

Using Julia v1.4 this works:

`model = Dense(W,b)`