# Building a Rolling Correlation Function

**URL:** <https://discourse.julialang.org/t/building-a-rolling-correlation-function/14862>\
**Category:** New to Julia\
**Created:** [September 12, 2018, 2:49pm UTC](https://discourse.julialang.org/t/building-a-rolling-correlation-function/14862 "2018-09-12T14:49:45Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![tnederlof](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@tnederlof](https://discourse.julialang.org/u/tnederlof)\
**Post date:** [September 12, 2018, 2:49pm UTC](https://discourse.julialang.org/t/building-a-rolling-correlation-function/14862/1 "2018-09-12T14:49:45Z")

</div>

As I often need to run a rolling correlation between two series I wanted to implement my own function and got most of the way there but I am running into a few issues I have been unsuccessful in solving.

**EDIT: The function now works with the explicit return**  
**EDIT2: The function now works with missing values**

1. This function currently either outputs the correlation number or if there is missing values puts a 0. How can I go about instead detecting if a missing value is in df\_sub and putting in the output vector a missing instead of 0?

```julia
using Pkg
using DataFrames

function roll_cor(df::DataFrame; n::Int64=126)::Array{Union{Float64, Missing}}
  cor_vec = Vector{Union{Float64, Missing}}(undef, (size(df)[1]))
  for i in n:size(df)[1]
    df_sub = df[(i-(n-1)):i,1:2]
    if !any(colwise(x -> any(ismissing.(x)), df_sub))
      cor_vec[i] = cor(convert(Array, df_sub))[1,2]
    else
      cor_vec[i] = missing
    end
  end
  return cor_vec
end

test = DataFrame(x = rand(2000), y = rand(2000))
roll_cor(test, n=126)

```

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [September 12, 2018, 3:20pm UTC](https://discourse.julialang.org/t/building-a-rolling-correlation-function/14862/2 "2018-09-12T15:20:43Z")

</div>

Have you tried using RollingFunctions.jl? I’ve used it with pretty good success. Even if it doesn’t directly take care of things for you, it is a good place to build from.

> **[GitHub - JeffreySarnoff/RollingFunctions.jl: Roll a window over data; apply a...](https://github.com/JeffreySarnoff/RollingFunctions.jl)**
>
> Roll a window over data; apply a function over the window. - GitHub - JeffreySarnoff/RollingFunctions.jl: Roll a window over data; apply a function over the window.

---

<div class="post-metadata">

**Author:** ![tnederlof](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@tnederlof](https://discourse.julialang.org/u/tnederlof)\
**Post date:** [September 12, 2018, 3:23pm UTC](https://discourse.julialang.org/t/building-a-rolling-correlation-function/14862/3 "2018-09-12T15:23:50Z")

</div>

Yes, I will look through the code there again though. Since there wasnt a correlation function in there as far as I could tell I figured building my own would help me learn. Its a good point that reviewing the structures used in that package could help.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [September 12, 2018, 3:35pm UTC](https://discourse.julialang.org/t/building-a-rolling-correlation-function/14862/4 "2018-09-12T15:35:58Z")

</div>

You can also pass in your own function. So, you could do something like

```julia
rolling(x->cor(x[:,1],x[:,2]),x,n)

```

where `x` is a `Tx2` matrix of your two series.

Additionally the Missings.jl package can help you propagate missing values, although I think `cor` automatically returns `missing` in this case on Julia 0.7 and above.

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [September 12, 2018, 3:41pm UTC](https://discourse.julialang.org/t/building-a-rolling-correlation-function/14862/5 "2018-09-12T15:41:22Z")

</div>

The function `roll_cor` has a return-type annotation `::Array{Float64}`, but it doesn’t actually return anything. You need `return cor_vec` or just `cor_vec` as the last line of your function. I suspect that is the convert error you’re seeing (although it will be easier to help if you can post the full error message).

---

<div class="post-metadata">

**Author:** ![tnederlof](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@tnederlof](https://discourse.julialang.org/u/tnederlof)\
**Post date:** [September 12, 2018, 4:05pm UTC](https://discourse.julialang.org/t/building-a-rolling-correlation-function/14862/6 "2018-09-12T16:05:17Z")

</div>

Thank you all fo the help. rdelts was right, I had to return the correlation array, the function was trying to convert some other part as the output since I omitted return.

Now that the function works properly for numbers (either puts a 0 or the correct correlation number), I am stuck trying to instead put either a missing or the correct correlation number. Currently I am using zeros to preallocate a vector of a certain length, is there an easy way to create a missing/Float64 vector of a certain length and then just populate that?

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [September 12, 2018, 4:31pm UTC](https://discourse.julialang.org/t/building-a-rolling-correlation-function/14862/7 "2018-09-12T16:31:17Z")

</div>

Sure, if you want a vector of either Float64 or Missing with length N, you can just do:

```julia
Vector{Union{Float64, Missing}}(undef, N)

```

where the `undef` tells Julia not to try to set each element of the array yet (since you’re going to set them yourself).

---

<div class="post-metadata">

**Author:** ![tnederlof](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@tnederlof](https://discourse.julialang.org/u/tnederlof)\
**Post date:** [September 12, 2018, 4:37pm UTC](https://discourse.julialang.org/t/building-a-rolling-correlation-function/14862/8 "2018-09-12T16:37:52Z")

</div>

Thank you that is exactly what I was searching for. So if I am understanding this right… undef tells Julia that each element may be a missing or a Float64 but dont worry about that until one of the two is set?

I updated the code in my original post to show what I am using now.

One thing I noticed is the speed is reduced roughly 4x slower (although still pretty fast in an absolute sense), is this just the realities of using a Union{Float64, Missing} versus just using Float64, where the compiler knows it only can be a float?

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [September 12, 2018, 4:57pm UTC](https://discourse.julialang.org/t/building-a-rolling-correlation-function/14862/9 "2018-09-12T16:57:06Z")

</div>

> [@tnederlof](#):
>
> Thank you that is exactly what I was searching for. So if I am understanding this right… undef tells Julia that each element may be a missing or a Float64 but dont worry about that until one of the two is set?

Almost, but not exactly. There are two steps involved in constructing a vector: (1) allocating space in memory and (2) actually setting that memory to some value. `Vector{T}(undef, N)` tells Julia to only do step (1), which saves some time. That means that until you actually set each element of the vector, the data it contains is _totally arbitrary_ and could be zeros or garbage or anything in between. Note how I get different contents each time I run this, based on whatever leftover data happened to be in memory:

```julia
julia> Vector{Float64}(undef, 2)
2-element Array{Float64,1}:
 6.92131585940106e-310
 0.0                  

julia> Vector{Float64}(undef, 2)
2-element Array{Float64,1}:
 6.92131532920064e-310
 6.9213154712449e-310 

julia> Vector{Float64}(undef, 2)
2-element Array{Float64,1}:
 5.0e-324
 0.0 

```

From this, you might be able to tell that there’s a bug in your code above. You are allocating an `undef` vector for `cor_vec`, but you’re only setting elements `n:size(df)[1]`. That means that the first `1:(n-1)` elements will have garbage contents. To actually initialize those values to zero, you can do:

```julia
cor_vec = Vector{....
cor_vec .= 0 # fill with zeros

```

or

```julia
cor_vec = Vector{...
cor_vec .= missing # fill with missing

```

As a shortcut for the above, you can also just use `zeros` as you did before:

```julia
julia> zeros(Union{Float64, Missing}, 5)
5-element Array{Union{Missing, Float64},1}:
 0.0
 0.0
 0.0
 0.0
 0.0

```

As a further note, the `undef` argument is new in Julia 0.7, but the behavior is not. Previously, `Vector{T}(N)` did exactly what `Vector{T}(undef, N)` does now, it was just less obvious. The `undef` serves as a marker to be careful (and remember to actually set each element of the vector after you’ve constructed it), and also as a slot to put other kinds of initialization behaviors. For example, we can now do:

```julia
julia> Matrix{Float64}(undef, 2, 2)
2×2 Array{Float64,2}:
 6.92132e-310 6.92132e-310
 6.92132e-310 0.0       

```

to allocate a Matrix without setting its contents, or we can use `I` instead of `undef` to do:

```julia
julia> using LinearAlgebra

julia> Matrix{Float64}(I, 2, 2)
2×2 Array{Float64,2}:
 1.0 0.0
 0.0 1.0

```

and create a Matrix and set its contents to be an identity matrix.

---

<div class="post-metadata">

**Author:** ![tnederlof](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@tnederlof](https://discourse.julialang.org/u/tnederlof)\
**Post date:** [September 12, 2018, 5:13pm UTC](https://discourse.julialang.org/t/building-a-rolling-correlation-function/14862/10 "2018-09-12T17:13:50Z")

</div>

This was really helpful for me to learn, thank you for spending the time creating the examples too. I read through the docs but these points didnt quite stick in my head.