Update – using latest Flux. I’d like to optimize a function of the singular values of a matrix. The Flux training loop barfs. Here’s my code using Flux, ForwardDiff, LinearAlgebra, GenericLinearAlgebra m = 20 X = randn(m,m) model = Dense(m,m) loss(X) = sum(GenericLinearAlgebra.svdvals(model(X))) …

Here’s how you can get the gradient working: using Flux, Zygote, ForwardDiff using GenericLinearAlgebra: svdvals svdvals2(X) = Zygote.forwarddiff(svdvals, X) m = 20 X = randn(m,m) model = Dense(m,m) loss(X) = sum(svdvals2(X)) ## nuclear norm loss(X) gradient(loss, X) This plugs in forwarddiff …

Welcome to this forum! See this post [image] Please read: make it easier to help you Meta Discussion Welcome to the Julia Discourse! We are enthusiastic about helping Julia programmers, both beginner and experienced. This public service announcement (PSA) outlines best pra…

It appears that you are using a very old version of Flux, later versions do not use TrackedArrays anymore. Maybe you could try to update Flux to the latest version?

@MikeInnes that works great. The Zygote.forwarddiff(svdvals,X) doesn’t do what one initially thinks it does (it seems to evaluate svdvals(X) instead of diff(svdvals(X)) ). Perhaps needs a better name – Zygote.evalf(svdvals,X) or Zygote.map(svdvals,X) ? Just a thought :slight_smile:

forwarddiff(f, x) is a little weird because it is the same as f(x) for the forward pass – it only affects how gradients are calculated, ie using forward rather than reverse mode. I’d love to have a name that gets that idea across better, although it might be fundamentally unintuitive until you’re e…

aah I see… that is an idea i haven’t been exposed to but seems like a great name could also expose the user to that idea (if they wanted to know more) how about map4autodiff, map4forwardpass, map4backpass ?

Using Flux to optimize a function of the Singular Values

Specific Domains Optimization (Mathematical)

baggepinnen May 18, 2020, 10:57am 2

Welcome to this forum! See this post

to make the code more readable

Topic		Replies	Views
Need help getting started with FluxML Machine Learning flux	5	1050	November 25, 2019
Unable to calculate the gradient of opnorm/svdvals/svd on gpu? General Usage question	9	180	June 13, 2025
Implementation of Spectral Normalization for Machine Learning General Usage	7	879	February 9, 2022
Help with gradient Machine Learning flux	6	1241	January 31, 2020
Flux differentiation error Machine Learning zygote	19	1799	November 19, 2020

Using Flux to optimize a function of the Singular Values

Related topics