# How to enforce weight matrix to be of a certain form (train a subset) in Flux?

**URL:** <https://discourse.julialang.org/t/how-to-enforce-weight-matrix-to-be-of-a-certain-form-train-a-subset-in-flux/33197>\
**Category:** Machine Learning\
**Tags:** question, flux\
**Created:** [January 10, 2020, 2:51pm UTC](https://discourse.julialang.org/t/how-to-enforce-weight-matrix-to-be-of-a-certain-form-train-a-subset-in-flux/33197 "2020-01-10T14:51:17Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![homocomputeris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/homocomputeris/32/8933_2.png) [@homocomputeris](https://discourse.julialang.org/u/homocomputeris)\
**Post date:** [January 10, 2020, 2:51pm UTC](https://discourse.julialang.org/t/how-to-enforce-weight-matrix-to-be-of-a-certain-form-train-a-subset-in-flux/33197/1 "2020-01-10T14:51:17Z")

</div>

For example, I want a layer specified by a symmetric or a tridiagonal matrix. Obviously, `train!` does not know about it, and an error is raised:  
`ArgumentError: cannot set entry (3, 1) off the tridiagonal band to a nonzero value (0.03524077074068107)`.

How to tell Flux to train only a certain subset of parameters (diagonals/upper half, etc)?

---

<div class="post-metadata">

**Author:** ![christos](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christos/32/7656_2.png) [@christos](https://discourse.julialang.org/u/christos)\
**Post date:** [July 3, 2020, 6:32am UTC](https://discourse.julialang.org/t/how-to-enforce-weight-matrix-to-be-of-a-certain-form-train-a-subset-in-flux/33197/2 "2020-07-03T06:32:49Z")

</div>

Hi, I have to deal with the same type of problem. Did you manage to find a way to do the update?

---

<div class="post-metadata">

**Author:** ![churchofthought](https://avatars.discourse-cdn.com/v4/letter/c/73ab20/32.png) [@churchofthought](https://discourse.julialang.org/u/churchofthought)\
**Post date:** [February 26, 2021, 3:54am UTC](https://discourse.julialang.org/t/how-to-enforce-weight-matrix-to-be-of-a-certain-form-train-a-subset-in-flux/33197/3 "2021-02-26T03:54:42Z")

</div>

I’m also having to deal with the same problem. I’m trying to do border-only convolutions. Their bulk is filled with zeros which should not be trainable.

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [February 26, 2021, 8:13am UTC](https://discourse.julialang.org/t/how-to-enforce-weight-matrix-to-be-of-a-certain-form-train-a-subset-in-flux/33197/4 "2021-02-26T08:13:57Z")

</div>

One strategy I can think of when it comes to dealing with these types of hard constraints is to make a layer which has only the paramters one wants to train and then “create” the full parameter in the forward pass in a way which Zygote agrees (maybe using `@nograd`).

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [February 26, 2021, 8:14am UTC](https://discourse.julialang.org/t/how-to-enforce-weight-matrix-to-be-of-a-certain-form-train-a-subset-in-flux/33197/5 "2021-02-26T08:14:22Z")

</div>

Using a custom loop and zeroing the gradient of elements you don’t want to change.  
[https://fluxml.ai/Flux.jl/stable/training/training/#Custom-Training-loops-1](https://fluxml.ai/Flux.jl/stable/training/training/#Custom-Training-loops-1)

---

<div class="post-metadata">

**Author:** ![darsnack](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/darsnack/32/10144_2.png) [@darsnack](https://discourse.julialang.org/u/darsnack)\
**Post date:** [February 26, 2021, 3:02pm UTC](https://discourse.julialang.org/t/how-to-enforce-weight-matrix-to-be-of-a-certain-form-train-a-subset-in-flux/33197/6 "2021-02-26T15:02:03Z")

</div>

> I’m also having to deal with the same problem. I’m trying to do border-only convolutions. Their bulk is filled with zeros which should not be trainable.

Sounds like this would benefit from a MaskedArrays.jl package that I’ve been thinking about writing for the purpose of pruning methods. Seems like an array type is the most elegant way to handle this.

More motivation to start prototyping.