# How to handle/ignore missing values when fitting differential equations?

**URL:** <https://discourse.julialang.org/t/how-to-handle-ignore-missing-values-when-fitting-differential-equations/43866>\
**Category:** General Usage\
**Tags:** diffeq, time-series\
**Created:** [July 29, 2020, 10:46am UTC](https://discourse.julialang.org/t/how-to-handle-ignore-missing-values-when-fitting-differential-equations/43866 "2020-07-29T10:46:26Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Irene](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/irene/32/16648_2.png) [@Irene](https://discourse.julialang.org/u/Irene)\
**Post date:** [July 29, 2020, 10:46am UTC](https://discourse.julialang.org/t/how-to-handle-ignore-missing-values-when-fitting-differential-equations/43866/1 "2020-07-29T10:46:26Z")

</div>

Hi,

I have a time-series of hourly weather data records that [you can check here](https://github.com/Irene-GM/data/blob/master/Hourly_Weather.csv). You can see that for every hour in these two days (July 16-17 2018), we have a measurement of temperature, and other variables. I am interested in using Julia to fit differential equations to my data, so I can assess whether this set up would work to forecast weather in the short-term ([like in this example](https://sebastiancallh.github.io/post/neural-ode-weather-forecast/)). However, as it happens with real data, you can see that **there are some missing values** in the CSV, that is, for a particular hour, there are no measurements taken.

I was wondering whether the differential equations library can actually work with gaps in the time-series, or it is absolutely mandatory to provide a value(s) at each time step (i.e. each hour in this case). I could impute or interpolate the missing values, but these would not be the real ones and that could be misleading during the fitting process.

Is it possible to pass data with gaps in such a set up?  
Can you provide any hint, if any, on how to do this?

Thanks for your help!

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 29, 2020, 12:51pm UTC](https://discourse.julialang.org/t/how-to-handle-ignore-missing-values-when-fitting-differential-equations/43866/2 "2020-07-29T12:51:37Z")

</div>

> [@Irene](#):
>
> I was wondering whether the differential equations library can actually work with gaps in the time-series, or it is absolutely mandatory to provide a value(s) at each time step (i.e. each hour in this case). I could impute or interpolate the missing values, but these would not be the real ones and that could be misleading during the fitting process.

Assuming you’re using DiffEqFlux, the missings should only show up in the data in the loss value that you write, so you just need to be careful there. For example, subtract the solution from your data and you’ll get missings, and then drop missings because you do `sum(abs2,x)` for sum squared error and it would do what you’re looking for on that kind of loss. For other losses you’d do similar missing dropping after doing the subtraction etc. against data.

---

<div class="post-metadata">

**Author:** ![Irene](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/irene/32/16648_2.png) [@Irene](https://discourse.julialang.org/u/Irene)\
**Post date:** [July 30, 2020, 6:45am UTC](https://discourse.julialang.org/t/how-to-handle-ignore-missing-values-when-fitting-differential-equations/43866/3 "2020-07-30T06:45:40Z")

</div>

Hi Chris,

Thanks for your reply! Ok, so it seems there is a way for `DiffEqFlux` to proceed with the fitting regardless of the missing values. Since I am very new to `Julia`+`DiffEqFlux`, I have a couple of extra questions, just to have it clear in my mind:

- **Loss function:** If I got this correctly, the key would be defining a loss function that can handle these ‘missing’ or ‘NaN’ for a given timestamp. In [this example from DiffEqFlux](https://juliahub.com/docs/DiffEqFlux/BdO4p/1.9.0/#Training-a-Neural-Ordinary-Differential-Equation), a L2 loss function is defined as follows:

```julia
function loss_n_ode(p)
    pred = predict_n_ode(p)
    loss = sum(abs2,ode_data .- pred) # L2 loss, I guess
    loss,pred
end

```

So in the event that this function receives a `missing` / `Nan`, and we add the robust loss function you kindly suggested, it could be rewritten as:

```julia
function loss_n_ode(p)
    clean = filter(!isnan, p) <--
    pred = predict_n_ode(clean) <--
    loss = sum(abs2, pred) <--
    loss,pred
end

```

So this would not crash in the event one of my [weather values](https://github.com/Irene-GM/data/blob/master/Hourly_Weather.csv) is `NaN`.

- **Delete empty rows of DataFrame?** By writing the previous item, I realized that the missing value is also an empty row in my DataFrame: **a gap in the temporal axis**. Maybe the function loss\_n\_node should not receive any NaN, because this row should be removed. Thus, in the event that I remove all [these empty rows](https://github.com/Irene-GM/data/blob/master/Hourly_Weather.csv) from my DataFrame, would still be ok to use DiffEqFlux with **an irregularly-sampled temporal axis**? I thought the temporal axis had to be “segmented” in equal units, but perhaps I am wrong on that.

Could you provide any hint/advice about this?  
Sorry for the lengthy message, as you can see I have a lot to learn and reflect! 😀  
Thanks a lot for your help!

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 3, 2020, 4:58am UTC](https://discourse.julialang.org/t/how-to-handle-ignore-missing-values-when-fitting-differential-equations/43866/4 "2020-08-03T04:58:21Z")

</div>

Sorry, JuliaCon got me behind.

> [@Irene](#):
>
> So this would not crash in the event one of my [weather values](https://github.com/Irene-GM/data/blob/master/Hourly_Weather.csv) is `NaN` .

Yup that should be good.

> [@Irene](#):
>
> **Delete empty rows of DataFrame?** By writing the previous item, I realized that the missing value is also an empty row in my DataFrame: **a gap in the temporal axis**. Maybe the function loss\_n\_node should not receive any NaN, because this row should be removed. Thus, in the event that I remove all [these empty rows](https://github.com/Irene-GM/data/blob/master/Hourly_Weather.csv) from my DataFrame, would still be ok to use DiffEqFlux with **an irregularly-sampled temporal axis**? I thought the temporal axis had to be “segmented” in equal units, but perhaps I am wrong on that.

It should be fine with DiffEqFlux, I just don’t know all of the DataFrame commands so someone else might need to help there, but you might want to do this in the data iterator since this kind of data mangling shouldn’t need to be differentiated.