@ChrisRackauckas apologies for the confusion. I realized that the control input was not causing the latency issue, so I simplified the model to just be a simple neural ODE that accepts an initial condition, no control.
I am however very concerned with the following post Composing a Neural ODE with another Neural Network
which seems to show a huge slowdown when I optimize a neural ODE composed with a neural network. This is the main problem I am trying to resolve.