How to enforce weight matrix to be of a certain form (train a subset) in Flux?

For example, I want a layer specified by a symmetric or a tridiagonal matrix. Obviously, train! does not know about it, and an error is raised:
ArgumentError: cannot set entry (3, 1) off the tridiagonal band to a nonzero value (0.03524077074068107).

How to tell Flux to train only a certain subset of parameters (diagonals/upper half, etc)?

Hi, I have to deal with the same type of problem. Did you manage to find a way to do the update?

I’m also having to deal with the same problem. I’m trying to do border-only convolutions. Their bulk is filled with zeros which should not be trainable.

One strategy I can think of when it comes to dealing with these types of hard constraints is to make a layer which has only the paramters one wants to train and then “create” the full parameter in the forward pass in a way which Zygote agrees (maybe using @nograd).

Using a custom loop and zeroing the gradient of elements you don’t want to change.
https://fluxml.ai/Flux.jl/stable/training/training/#Custom-Training-loops-1

I’m also having to deal with the same problem. I’m trying to do border-only convolutions. Their bulk is filled with zeros which should not be trainable.

Sounds like this would benefit from a MaskedArrays.jl package that I’ve been thinking about writing for the purpose of pruning methods. Seems like an array type is the most elegant way to handle this.

More motivation to start prototyping.