Flux.gradient throws MethodError even though loss is evaluated just fine

This is just the same question and answer as Using Flux: gradient on DifferentialEquations: solve results in an error and https://www.reddit.com/r/Julia/comments/u4qgxc/flux_gradient_cant_differentiate_a_function/.