I am having a problem speeding up my Neural ODE gradient computations. Namely, I have defined a neural ode struct which accepts a control input and am aiming to optimize the parameters of the neural network to optimize an objective. In the MRE posted below, I am finding that ForwardDiffSensitivity t…

[image] George_Stepaniants: am finding that ForwardDiffSensitivity takes approximately 5 seconds to compute the gradient of a loss where I simulate my neural ODE with 100 different control inputs. The cost of forward is dependent on the number of ODEs and parameters, not based on the number o…

My current neural network has upwards of 250 parameters so I assumed it was in the regime where adjoint methods should outperform forward diff. I’ve also benchmarked the different sensitivity methods against each other just on one control input. Perhaps something in my code needs to be optimized? W…

Another issue, if I want to parallelize the looping over controls with @Threads.threads like cd(@__DIR__) using Pkg Pkg.activate(".") using Lux using ComponentArrays using Zygote using SciMLSensitivity using ForwardDiff using DifferentialEquations using Optimization using OptimizationOptimJL usin…

[image] George_Stepaniants: Would making sure my code is type-stable help here? That always helps and is the starting point. If you do a flame graph of the forward pass with no differentiation, what are the hot spots?

@ChrisRackauckas from just a forward pass (no differentiation), I get a flamegraph which looks like this: https://drive.google.com/drive/folders/1QQcPh8FajL_xNokrpdf_ND-Jm6Bt4iFm?usp=sharing It doesn’t seem like there is one operation that is taking up all of the time.

Is this the right benchmark? It shows a NeuralODE call taking up most of the time, but you don’t have any in your code above. And I don’t see any interpolations from the controls, so it doesn’t look like it’s measuring the code you’re showing. Also no multithreading is shown. Tthe selu is taking up…

@ChrisRackauckas apologies for the confusion. I realized that the control input was not causing the latency issue, so I simplified the model to just be a simple neural ODE that accepts an initial condition, no control. I am however very concerned with the following post Composing a Neural ODE with …

ForwardDiffSensitivity Faster than Adjoint Methods

General Usage

ChrisRackauckas August 10, 2023, 4:08pm 8

Reading your other post, I think you sent the flamegraph from this model Composing a Neural ODE with another Neural Network

Topic		Replies	Views
All Adjoint Methods Fail on Simple Neural ODE Example General Usage	7	303	July 22, 2023
Optimizating computational time of gradient on big linear UDEs Modelling & Simulations	11	473	January 11, 2024
Composing a Neural ODE with another Neural Network General Usage	2	423	August 17, 2023
Neural ODE fitting really slow Machine Learning diffeq , flux , sciml	13	2098	September 11, 2020
Discrepancy of ODE Sensitivity Analysis paper results with benchmarks Numerics benchmark , ode , sciml	31	569	June 18, 2025

ForwardDiffSensitivity Faster than Adjoint Methods

Related topics