# Nested Task error: Bad window span - what?

**URL:** <https://discourse.julialang.org/t/nested-task-error-bad-window-span-what/67959>\
**Category:** Data\
**Tags:** question, dataframes\
**Created:** [September 9, 2021, 9:57pm UTC](https://discourse.julialang.org/t/nested-task-error-bad-window-span-what/67959 "2021-09-09T21:57:45Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![SubTer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/subter/32/28830_2.png) [@SubTer](https://discourse.julialang.org/u/SubTer)\
**Post date:** [September 9, 2021, 9:57pm UTC](https://discourse.julialang.org/t/nested-task-error-bad-window-span-what/67959/1 "2021-09-09T21:57:45Z")

</div>

hi folks,  
I am trying to add a column to a df i already have populated, and on a smaller batch size it works just fine, example code below:

```julia
Timeddf = @linq dfData |>
          groupby(:ID) |>
          transform(Trailing12 = running(prod, (:Growth .+1), 12).-1) 

```

using Dataframes & Rollingfunctions, so when the dfData is about 2000 rows, no issues, code executes and does what is expected, but when I add all of the IDs, dfData grows to about 100k rows, and the above code throws out an error:

```julia
nested task error: 
        Bad window span (12) for length 10.

```

which seems to originate in splitapplycombine.jl part of Dataframes package, but I cant quite figure out why - is it some soft of memory limit I am hitting with how dataframes can be used in Julia?

Thanks!

edit: issue seems to happen only when df gets over 8000 rows or so, and then the error starts showing up…

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [September 10, 2021, 1:16am UTC](https://discourse.julialang.org/t/nested-task-error-bad-window-span-what/67959/2 "2021-09-10T01:16:14Z")

</div>

Could you post some runnable code so people can reproduce this on their own machines? Code to make some fake data in `dfData` and to include all the modules you’re using would be ideal.

---

<div class="post-metadata">

**Author:** ![SubTer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/subter/32/28830_2.png) [@SubTer](https://discourse.julialang.org/u/SubTer)\
**Post date:** [September 10, 2021, 2:15am UTC](https://discourse.julialang.org/t/nested-task-error-bad-window-span-what/67959/3 "2021-09-10T02:15:10Z")

</div>

struggling to reproduce the error with random data df…

using DataFrames, Query, and RollingFunctions packages  
Query for the linq function there to read the df, DataFrames to actually store a query as a df, and RollingFunctions package for the “running” method.

the below works just fine unfortunately, so starting to think this isnt a pure size issue…

```julia
bigDf = DataFrame(i = 1:1000000, Growth = rand(1000000), Categories = rand(["a", "b", "c", "d", "f", "g"], 1000000))

TimedDfTest = @linq bigDf |>
         groupby(:Categories) |>
         transform(Trailing12 = running(prod, (:Growth .+1), 12).-1)

```

yet when using actual data in similar way, it fails whenever the df grows over ~8000 rows. And spits out that error message about Bad window span.

---

<div class="post-metadata">

**Author:** ![SubTer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/subter/32/28830_2.png) [@SubTer](https://discourse.julialang.org/u/SubTer)\
**Post date:** [September 10, 2021, 2:32am UTC](https://discourse.julialang.org/t/nested-task-error-bad-window-span-what/67959/4 "2021-09-10T02:32:56Z")

</div>

Think i know whats going on…

```julia
Timeddf = @linq dfData |>
          groupby(:ID) |>
          transform(Trailing12 = running(prod, (:Growth .+1), 12).-1) 

```

the 12 in there, there are ID categories that only have up to 10 rows, so it gets broken the moment it tries to do 12 passes where only 10 rows exist for the groupby category.  
Hmmm… is there a simple way to tell it to ignore those? alternatively I guess i can filter out original df to ensure those dont occur.

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [September 13, 2021, 7:26pm UTC](https://discourse.julialang.org/t/nested-task-error-bad-window-span-what/67959/5 "2021-09-13T19:26:46Z")

</div>

try this way (I don’t know if simple and efficient)

the scheme used [here](https://discourse.julialang.org/t/how-to-compute-a-cumulative-in-a-dataframe-without-a-for-loop/2156/44)

```julia

using ShiftedArrays, DataFrames,RollingFunctions, Query

bigDf = DataFrame(i = 1:100, Growth = rand(100), Categories = rand(["a", "b", "c", "d", "f", "g"], 100))

TimedDfTest = @linq bigDf |>
         groupby(:Categories) |>
         transform(Trailing = running(prod, (:Growth .+1), 3).-1)

runshp(arr, sh) =cumprod(arr)./ShiftedArray(cumprod(arr),sh,default=1)
       
withOnlySA=transform(groupby(bigDf, :Categories),:Growth => (x -> runshp(x.+1, 3).-1) => :Trailing)       

```

considera che

```julia

julia> withOnlySA.Trailing ≈ TimedDfTest.Trailing
true

julia> runshp([1,2,3,4],5)
4-element Vector{Float64}:
  1.0
  2.0
  6.0
 24.0

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [September 13, 2021, 7:33pm UTC](https://discourse.julialang.org/t/nested-task-error-bad-window-span-what/67959/6 "2021-09-13T19:33:28Z")

</div>

You could just say

```julia
Trailing12 = begin 
    if length(:Growth) > 12 
        fill(missing, length(:Growth))
    else 
        running(prod, ...)
    end
end

```

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [September 13, 2021, 8:25pm UTC](https://discourse.julialang.org/t/nested-task-error-bad-window-span-what/67959/7 "2021-09-13T20:25:11Z")

</div>

for BigDf is better this:

```julia

runshp1(arr,sh)=prod.(getindex.([arr],(:).(ShiftedArray(1:length(arr),sh-1,default=1),1:length(arr))))

withOnlySA=transform(groupby(bigDf, :Categories),:Growth => (x -> runshp1(x.+1, 3).-1) => :Trailing)       

```