# Constrained weight matrices in Flux

**URL:** <https://discourse.julialang.org/t/constrained-weight-matrices-in-flux/42974>\
**Category:** Machine Learning\
**Tags:** flux\
**Created:** [July 13, 2020, 8:24am UTC](https://discourse.julialang.org/t/constrained-weight-matrices-in-flux/42974 "2020-07-13T08:24:08Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![grero](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/grero/32/109_2.png) [@grero](https://discourse.julialang.org/u/grero)\
**Post date:** [July 13, 2020, 8:24am UTC](https://discourse.julialang.org/t/constrained-weight-matrices-in-flux/42974/1 "2020-07-13T08:24:08Z")

</div>

Is there a way to constrain the weight matrices trained by Flux? I’m interested in using Flux to train neural networks where I’d like to e.g. specify the proportion of negative and positive weights, which would simulate units that are consistently either excitatory or inhibitory. I would also like to explore low-rank weight matrices, i.e. where the matrices are combinations of a few linearly independent components. Is this achievable with Flux?

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [July 13, 2020, 12:43pm UTC](https://discourse.julialang.org/t/constrained-weight-matrices-in-flux/42974/2 "2020-07-13T12:43:26Z")

</div>

I think it’s much easier to parameterize than to constrain. For example, let’s say you want to make a `LowRankLinear` layer. Then you would do:

```julia
julia> struct LowRankLinear{S, T}
           w1::S
           w2::T
       end

julia> function LowRankLinear(m::Integer, n::Integer; rank, init=rand)
           w1 = init(rank, m)
           w2 = init(n, rank)
           LowRankLinear(w1, w2)
       end
LowRankLinear

julia> (l::LowRankLinear)(x) = l.w2 * l.w1 * x

julia> LowRankLinear(10, 20, rank=4)(rand(10, 32))
20×32 Array{Float64,2}:
...

```

Note that instead of computing things directly, if your original model is more complex, you could build it on the fly in the forward loop using `l.w2 * l.w1` (i.e., you low-rank weight matrix).

(Unrelated, for “copy-pastability” I put `rand` as default initializer, Flux generally uses `glorot_uniform`.)