# Unexpected behavior when using Impute.locf within @by

**URL:** <https://discourse.julialang.org/t/unexpected-behavior-when-using-impute-locf-within-by/105642>\
**Category:** Data\
**Tags:** dataframes, dataframesmeta\
**Created:** [November 1, 2023, 6:25am UTC](https://discourse.julialang.org/t/unexpected-behavior-when-using-impute-locf-within-by/105642 "2023-11-01T06:25:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![BeitianMa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/beitianma/32/43745_2.png) [@BeitianMa](https://discourse.julialang.org/u/BeitianMa)\
**Post date:** [November 1, 2023, 6:25am UTC](https://discourse.julialang.org/t/unexpected-behavior-when-using-impute-locf-within-by/105642/1 "2023-11-01T06:25:07Z")

</div>

As the following code shows, i want to forward fill missing values use Impute.locf function, but just within the same :id

```julia
using DataFramesMeta, Impute

df = DataFrame(id = repeat(1:3, 2), value = [1,missing,3,4,missing,missing])

df = @chain df begin
    @by(:id, :value = Impute.locf(:value),	$(:))
end

# following code raises error too, so this doesn't seem to be DataFramesMeta's problem
# combine(groupby(df, :id), :value => (x -> Impute.locf(x)))

```

Unexpectedly, it raises

```julia
ERROR: AssertionError: !(all(ismissing, data))

```

this is clearly beacause there are all missing value under the same :id=2, but the following code

```julia
df = DataFrame(id = repeat(1:3, 2), value = [missing,missing,missing,missing,missing,missing])

df = @chain df begin
    @transform(:value = Impute.locf(:value))
end

```

completed with no error. It just leaves all values missing, which is the desired result

```julia
Row │ id value   
     │ Int64 Missing 
─────┼────────────────
   1 │ 1 missing 
   2 │ 2 missing 
   3 │ 3 missing 
   4 │ 1 missing 
   5 │ 2 missing 
   6 │ 3 missing

```

My questions are:

1. Is it a bug or a feature (for some concerns I don’t know)?
2. How do I get the (grouped) results? Of course, the simpler the code, the better.

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [November 1, 2023, 12:26pm UTC](https://discourse.julialang.org/t/unexpected-behavior-when-using-impute-locf-within-by/105642/2 "2023-11-01T12:26:07Z")

</div>

Group `2` has only `missing` values so `Impute.locf` errors as there is no value that can be used for filling the data.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [November 1, 2023, 3:12pm UTC](https://discourse.julialang.org/t/unexpected-behavior-when-using-impute-locf-within-by/105642/3 "2023-11-01T15:12:24Z")

</div>

I think the subtlety that OP is stumbling over is this:

```julia
julia> locf([missing])
1-element Vector{Missing}:
 missing

julia> locf(Union{Float64, Missing}[missing])
ERROR: AssertionError: !(all(ismissing, data))

```

So when a vector is all `missing` whether or not the imputation works depends on the type of the vector.

You can work around this by narrowing the type of the sub-vectors (although this is probably not great performance-wise):

```julia
julia> combine(groupby(df2, :id), :value => (x -> locf(identity.(x))))
6×2 DataFrame
 Row │ id value_function
     │ Int64 Int64?
─────┼───────────────────────
   1 │ 1 1
   2 │ 1 4
   3 │ 2 missing
   4 │ 2 missing
   5 │ 3 3
   6 │ 3 3

```