MethodError: no method matching getrf! when I try to use GPU to training a universal differential equation with ForwardDiff.jl

Specific Domains Optimization (Mathematical)
1

Hi, thank you for your attention. So, I’m trying to make use of GPU to train a universal differential equation but, when I run the code to make the optimization with Optimization.jl I got this error below. I already tried to search for similar problems but none of them helped me to solve mine. Someone had this issue?

MethodError: no method matching getrf!(::CuArray{ForwardDiff.Dual{ForwardDiff.Tag{ODEFunction{true, SciMLBase.AutoSpecialize, typeof(nn_dynamics!), UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Float32}, Float32, 11}, 2, CUDA.Mem.DeviceBuffer})
Closest candidates are:
  getrf!(::StridedCuMatrix{Float32}) at ~/.julia/packages/CUDA/DfvRa/lib/cusolver/dense.jl:120
  getrf!(::StridedCuMatrix{Float64}) at ~/.julia/packages/CUDA/DfvRa/lib/cusolver/dense.jl:120
  getrf!(::StridedCuMatrix{ComplexF32}) at ~/.julia/packages/CUDA/DfvRa/lib/cusolver/dense.jl:120
  ...
2

If you use Zygote you shouldn’t hit this. You can open an issue in CUDA.jl to implement a generic getrf! kernel, but just avoiding the need for a generic kernel is probably a better way here.

1 Like
3

Thank you so much for your answear. I changed the AD type to AutoZygote and I had to change the sensealg to QuadratureAdjoint when calling solve function. Well, now I am facing the Performing scalar indexing on task Task problem, even though I used the CUDA.allowscalar() do ..... end on the ODE function. Maybe I will have to spend some time vectorizing the operations. Do you know any tutorial about using GPU to train UDE model (mechanistic model coupled with neural network)?

4

I haven’t found a case where it was actually beneficial do GPU the NN because they usually end up too small, so we went the other direction and created SimpleChains.jl for a lot of the UDE use cases.

The climate and Fisher-KPP examples did have GPU codes at one time though,

not quite sure where those went, but I need to update those codes to proper tutorials anyways (“soon”). But even then, the NN wasn’t really the bottleneck in those cases to the point where GPU kernel speed matters. The issue is that if the other equations are not highly structured (like a PDE), then the prior knowledge portion needs to be done on CPU (because then it’s not good structured for SIMD), and so you have some form of data transfer in the steps, which then makes the cutoff point a lot higher.

That said, UDE on a PDE with a ROCK method is probably the best place to find an example where it works out.

5

Thank you for your reply. I will take a look at the links you sent, I’m very grateful for that. In my case I use ODE coupled with NN, my setup is an ODE with 11 states or chemical species and an NN with 11 inputs and 7 outputs but this model is just a cut out of a big one and I’m trying to see if the NN would predict the missing signals from the cutout model. So, my goal is to run the training with different initial vectors and I thought to use GPU to accelerate the experiment. But, I think I will let this for a moment and use just the CPU. If you have any advice, you are welcome. Thank you again :smiley:

6

Yeah that’s a CPU case.

1 Like