Dear All, I had a small pet project over the christmas, for some very specific application (steganography) I want to train a neural network while regularizing the gradient with respect to the input, which means that during training, I need to compute 2nd order gradient. I have chosen Lux.jl instea…

using Lux, CUDA, cuDNN, Random, OneHotArrays, Zygote using Functors, Optimisers, Printf model = Chain( Conv((5, 5), 1 => 6, relu), MeanPool((2, 2)), Conv((5, 5), 6 => 16, relu), MeanPool((2, 2)), FlattenLayer(3), Chain( Dense(256 => 128, relu), Dense(128 => 8…

Second order gradient with Lux, Zygote, CUDA, Enzyme

Specific Domains Machine Learning

avikpal December 31, 2024, 5:35pm 6

Lux swaps in ForwardDiff for nested AD with Zygote (Nested AD with Lux etc) and that will fail here without a ForwardDiff.Dual overload

Topic		Replies	Views
Lux (And Flux), "parallel" Network Input. When Input is flat, Zygote gradient works, when input is not flat it doesn't Machine Learning flux , zygote , lux	10	630	February 5, 2024
Lux, ComponentArrays and flat parameters : computing the gradient works with Zygote but not with Enzyme New to Julia enzyme	16	1522	May 14, 2024
[ANN] Lux.jl: Explicitly Parameterized Neural Networks in Julia Package Announcements package , announcement , machine-learning	50	10922	April 27, 2024
Zygote gradient error with `reduce` on GPU Machine Learning cuda , zygote	3	302	February 6, 2023
Fast Hessian and Gradient for PINNS using Enzyme/Zygote Performance question , flux , zygote , enzyme , hessian	0	346	July 23, 2023

Second order gradient with Lux, Zygote, CUDA, Enzyme

Related topics