Is it possible to use a custom constrained optimizer or to modify the gradients to only allow for positive weights and biases?
I want to say this has been discussed before on Discourse, but not being able to find a thread I’d suggest looking at Feature request: Modifying Dense Layer to accommodate kernel/bias constraints and kernel/bias regularisation · Issue #1389 · FluxML/Flux.jl · GitHub. Note that we are moving away from this “implicit”
Params model into storing both model weights and gradients in proper structures. If you’d like something more futureproof, have a look at Home · Optimisers.jl.
Thanks. I tried the regularization path but I couldn’t make it work, I am looking into the custom optimizer now
An alternative approach is to introduce an activation function for the weights. That way the weights “as applied” are non-negative, but they posses a real valued learnable “latent state”.
using Flux m = Dense(3, 2, relu) x = rand(Float32, 3, 5) (a::Dense)(x::AbstractVecOrMat, g) = a.σ.(g.(a.weight)*x .+ g.(a.bias)) y1 = m(x) # normal forward pass y2 = m(x, relu) # forward pass with non-negative weights
Oooh that is a good idea, now maybe stupid question…
I don’t see any problem using a normal gradient descent, but do you think it would create some problemsusing a method requiring an Hessian (stochastic LBFGS) for training since it depends on the information on the previous gradient? (that was the main reason why I was looking if it was possible to have a constrained optimizer
I don’t think it should be an issue, but I might be missing something.
But in general nonnegative weights make a network a lot less expressive. You are limiting the weights to a single orthant. I think both approaches have drawbacks
Optimizer approach: How to efficiently project to the feasible domain.
weight activation approach: How to avoid dying or saturated synapses (ReLU and Sigmoid case respectively).