Multiple shooting and multiple options

[JordiBolibar]

I am trying to add multiple shooting to a neural differential equation. After browsing multiple options, I see I have two possibilities:

• Use DiffEqFlux.jl’s multiple_shoot
• Use DiffEqParamEstim.jl’s multiple_shooting_objective.

According to DiffEqFlux’s docs, this option “is a specialized form for implicit layer deep learning (known as data shooting) which assumes full observability of the underlying dynamics and lack of noise”.

I am unsure what is exactly meant by this. Right now, in our problem we are running an ODE forward as a ground truth and we are adding Fisher noise on top of it. Then, we are trying to learn the underlying dynamics from that “ground truth” using a neural ODE. Would DiffEqFlux be the way to go there, or it would only work for cases where the dynamics are perfectly clean and known? So far I have implemented this and it keeps going without even making it to the first epoch.

Would DiffEqParamEstim be a better alternative? I like the interface of multiple_shoot better, since it allows for more control.

If necessary, I can provide more details on the exact ODE and problem we are trying to solve. Thanks in advance!

[Vaibhavdixit02]

Prefer the DiffEqFlux one, that should be more recent

[ChrisRackauckas]

DiffEqFlux’s is a special form that I call data shooting. It’s not the more general form. I think someone should generalize it a bit before I’d recommend it more.

[JordiBolibar]

I see. What do you mean exactly by "it is a specialized form for implicit layer deep learning (known as data shooting) which assumes full observability of the underlying dynamics and lack of noise”? Do you need to have a dense continuous solution without noise for it to work?

We are trying it in our SphereUDE project with @facusapienza and it doesn’t seem to converge for a controlled experiment with noise.

[ChrisRackauckas]

@JordiBolibar

I see. What do you mean exactly by "it is a specialized form for implicit layer deep learning (known as data shooting) which assumes full observability of the underlying dynamics and lack of noise”? Do you need to have a dense continuous solution without noise for it to work?

No I mean it takes the data points directly as samples for initial conditions, rather than having free floating initial conditions that are learned. This thus assumes the data points are “exactly correct” in some sense.

@JordiBolibar

We are trying it in our SphereUDE project with @facusapienza and it doesn’t seem to converge for a controlled experiment with noise.

Not surprising.

[JordiBolibar]

OK, so what is the right tool for this? It’s really not clear in the docs.

[ChrisRackauckas]

I don’t have an implementation I’d prefer right now.

[facusapienza]

@ChrisRackauckas I am thinking for this problem just to make an implementation of multiple shooting where the initial conditions for each batch can be learned. I understand the example in multiple_shoot is just for the noiseless case, so no initial condition is trained. I am planning to implement something like this in here SphereUDE.jl by increasing the parameter to be optimized, in the same way we did this for the initial condition:

    # Set component vector for Optimization
    if params.train_initial_condition
        β = ComponentVector{Float64}(θ=θ, u0=params.u0)
    else
        β = ComponentVector{Float64}(θ=θ)
    end

I can try to work on an example that I can then PR to DiffEqFlux.jl if this is an useful contribution. I am happy to help here with an implementation for the case with noise, I just need some guidance of where to contribute and being sure this is a required feature.