# \`filter\` on a \`DataFrame\` is not allocating a new object as expected

**URL:** <https://discourse.julialang.org/t/filter-on-a-dataframe-is-not-allocating-a-new-object-as-expected/76447>\
**Category:** Data\
**Tags:** dataframes\
**Created:** [February 14, 2022, 5:29pm UTC](https://discourse.julialang.org/t/filter-on-a-dataframe-is-not-allocating-a-new-object-as-expected/76447 "2022-02-14T17:29:29Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![hdavid16](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hdavid16/32/11531_2.png) [@hdavid16](https://discourse.julialang.org/u/hdavid16)\
**Post date:** [February 14, 2022, 5:29pm UTC](https://discourse.julialang.org/t/filter-on-a-dataframe-is-not-allocating-a-new-object-as-expected/76447/1 "2022-02-14T17:29:29Z")

</div>

Hi,

I just attempted the following and was surprised that my original table was modified. I would have expected `filter` to allocate a new `DataFrame` so that `setdiff!` would not have modified the original `DataFrame`. I also tried `view=false` (which is the default), but the original table is mutated. Is this a bug?

```julia
julia> gtable = DataFrame(A=[1,2,3],B=[[1],[2],[3]])
3×2 DataFrame
 Row │ A B
     │ Int64 Array…
─────┼───────────────
   1 │ 1 [1]
   2 │ 2 [2]
   3 │ 3 [3]

julia> b = filter(i -> i.A == 1, gtable).B[1]
1-element Vector{Int64}:
 1

julia> setdiff!(b,[1])
Int64[]

julia> gtable
3×2 DataFrame
 Row │ A B
     │ Int64 Array…
─────┼────────────────
   1 │ 1 Int64[]
   2 │ 2 [2]
   3 │ 3 [3]

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [February 14, 2022, 5:42pm UTC](https://discourse.julialang.org/t/filter-on-a-dataframe-is-not-allocating-a-new-object-as-expected/76447/2 "2022-02-14T17:42:38Z")

</div>

That’s an interesting observation. DataFrames uses `copy` for these sorts of things rather than `deepcopy`. I wouldn’t call this a bug, necessarily, but it is something to look out for.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [February 14, 2022, 7:11pm UTC](https://discourse.julialang.org/t/filter-on-a-dataframe-is-not-allocating-a-new-object-as-expected/76447/3 "2022-02-14T19:11:03Z")

</div>

This is expected. Base Julia works in exactly the same way.

---

<div class="post-metadata">

**Author:** ![hdavid16](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hdavid16/32/11531_2.png) [@hdavid16](https://discourse.julialang.org/u/hdavid16)\
**Post date:** [February 14, 2022, 8:07pm UTC](https://discourse.julialang.org/t/filter-on-a-dataframe-is-not-allocating-a-new-object-as-expected/76447/4 "2022-02-14T20:07:32Z")

</div>

Thank you for your responses. @bkamins, I see that this is indeed the behavior in Base Julia:

```julia
julia> x=[[1,2,3]]
1-element Vector{Vector{Int64}}:
 [1, 2, 3]

julia> y=copy(x)
1-element Vector{Vector{Int64}}:
 [1, 2, 3]

julia> setdiff!(y[1],[2])
2-element Vector{Int64}:
 1
 3

julia> x
1-element Vector{Vector{Int64}}:
 [1, 3]

```

Should there be an option to fully allocate a new DataFrame using deepcopy instead of copy?

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [February 14, 2022, 9:24pm UTC](https://discourse.julialang.org/t/filter-on-a-dataframe-is-not-allocating-a-new-object-as-expected/76447/5 "2022-02-14T21:24:18Z")

</div>

I guess we could allow passing a function to `copycols`. But what’s the use case exactly?

---

<div class="post-metadata">

**Author:** ![hdavid16](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hdavid16/32/11531_2.png) [@hdavid16](https://discourse.julialang.org/u/hdavid16)\
**Post date:** [February 14, 2022, 11:37pm UTC](https://discourse.julialang.org/t/filter-on-a-dataframe-is-not-allocating-a-new-object-as-expected/76447/6 "2022-02-14T23:37:54Z")

</div>

I am using it in simulation and am storing an initial state in the dataframe that gets updated. I guess, we don’t necessarily need to implement a `deepcopy` option, but the docstring could be updated to indicate that the new dataframe that is allocated is done by using `copy` rather than `deepcopy`.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [February 15, 2022, 7:00am UTC](https://discourse.julialang.org/t/filter-on-a-dataframe-is-not-allocating-a-new-object-as-expected/76447/7 "2022-02-15T07:00:38Z")

</div>

> [@hdavid16](#):
>
> but the docstring could be updated to indicate that the new dataframe that is allocated is done by using `copy` rather than `deepcopy` .

Everywhere in the docstring where `copycols` is supported we write that we copy columns (not deepcopy).

If you think clarifying this would be helpful can you please make a PR to the manual (not to the documentation since we have very many places where the same issue is present) in a place where you think it would be most useful for you? Thank you!

---

<div class="post-metadata">

**Author:** ![hdavid16](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hdavid16/32/11531_2.png) [@hdavid16](https://discourse.julialang.org/u/hdavid16)\
**Post date:** [February 19, 2022, 4:27am UTC](https://discourse.julialang.org/t/filter-on-a-dataframe-is-not-allocating-a-new-object-as-expected/76447/8 "2022-02-19T04:27:22Z")

</div>

Oh ok. I should check the manual then! Thanks @bkamins