# What if \`df.col .= v\` was in-place?

**URL:** <https://discourse.julialang.org/t/what-if-df-col-v-was-in-place/89367>\
**Category:** Data\
**Tags:** poll, dataframes\
**Created:** [October 27, 2022, 1:02pm UTC](https://discourse.julialang.org/t/what-if-df-col-v-was-in-place/89367 "2022-10-27T13:02:45Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![gustafsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gustafsson/32/3761_2.png) [@gustafsson](https://discourse.julialang.org/u/gustafsson)\
**Post date:** [October 27, 2022, 1:02pm UTC](https://discourse.julialang.org/t/what-if-df-col-v-was-in-place/89367/1 "2022-10-27T13:02:45Z")

</div>

You may or may not be surprised to learn that for a DataFrame `.=` isn’t always in-place. It surprised me for sure, but I didn’t follow the announcements so that’s on me.

What if it was always in-place for existing columns though? And what if normal assignment only created an alias if explicitly declared (with copy on assignment otherwise, with fill for scalars)? Would that be at all breaking?

I’m curious to learn if you have used or seen:

- `df.col .= v` on an existing column where you need it to not be in-place

- `df.col = v` where you need `v === df.col`

- Either of the above with `df[!, :col]`

Here’s a tentative implementation to run with your unit tests if you’d like to: [In-place broadcast assignment by gustafsson · Pull Request #3206 · JuliaData/DataFrames.jl · GitHub](https://github.com/JuliaData/DataFrames.jl/pull/3206)

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [October 27, 2022, 1:52pm UTC](https://discourse.julialang.org/t/what-if-df-col-v-was-in-place/89367/2 "2022-10-27T13:52:15Z")

</div>

I’m far from an expert DataFrames user but it seems unexpected to me that `df.col .= v` is not in-place.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 27, 2022, 1:57pm UTC](https://discourse.julialang.org/t/what-if-df-col-v-was-in-place/89367/3 "2022-10-27T13:57:30Z")

</div>

Here are two non-exhaustive reasons for the current behavior, as far as I understand it.

- DataFrames.jl wishes to allow `df.newcol .= 1` and `df.newcol .= 1:3` to conveniently create new columns. Both of these _have_ to allocate. If we were to allow `df.existincol .= 1` to _not_ allocate, then there would be more complicated rules for users to learn.
- DataFrames.jl doesn’t want new users to worry about conversion rules. if `df.x` is a `Vector{Int}`, we do not want `df.x .= 'a'` to auto-promote to `Int`.

There are multiple competing goals and DataFrames.jl chose a behavior that satisfied some constraints, but clearly don’t fall in line with everyone’s intuition.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [October 27, 2022, 2:35pm UTC](https://discourse.julialang.org/t/what-if-df-col-v-was-in-place/89367/4 "2022-10-27T14:35:01Z")

</div>

As this is a poll:

For the first point, it would be better to throw an error, consistent with Julia arrays syntax, when the array does not exist.

Regarding the second point, `df.x .= 'a'` does promote to `Int` (DataFrames v1.3.6).

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [October 27, 2022, 2:46pm UTC](https://discourse.julialang.org/t/what-if-df-col-v-was-in-place/89367/5 "2022-10-27T14:46:53Z")

</div>

> [@rafael.guerra](#):
>
> DataFrames v1.3.6

This is the point of this pool. In DataFrames.jl 1.4 it does not promote because this promotion was confusing users.

Similarly users expected that `df.x .= value` would work even if `:x` is not present in a data frame,  
exactly in the same way as `df[:, :x] = value` and similar work if `:x` is not present in a data frame. Writing `df.x = vector` or `df.x .= value` is AFAICT the most common way currently to add columns to a data frame.

Now, the point is that the design idea behind not in-place behavior of `df.x .= value` is that we wanted to make sure that this operation always produces the same value stored in `:x` column no matter if `:x` was previously present in `df` or not. Exactly like `df.x = vector` currently behaves (by replacing whatever is or is not present in column `:x` by `vector`).

Tomorrow I will write a longer blog post about the reasoning behind this design.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [October 27, 2022, 2:55pm UTC](https://discourse.julialang.org/t/what-if-df-col-v-was-in-place/89367/6 "2022-10-27T14:55:19Z")

</div>

> [@bkamins](#):
>
> df.x .= value

Shouldn’t the deviations from Julia’s base syntax be decorated with macros?

A fake example for a new `df.x` column about to be created:  
`@df df.x .= value`

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [October 27, 2022, 4:35pm UTC](https://discourse.julialang.org/t/what-if-df-col-v-was-in-place/89367/7 "2022-10-27T16:35:02Z")

</div>

From the docs

> To get a copy of the column you can use german[:, :Sex] or german[:, “Sex”]. In this case changing the vector returned by this operation does not affect the data stored in the german data frame.

Reading this, I now find the following behavior confusing

```julia
df = DataFrame(:A => [1,2,3,4])
df[:, :A] .= 5
df[!, :A] == [5, 5, 5, 5] #true

```

Is it intended that `df[:, col]` returns a copy on RHS but not on LHS? It seems to me the desired behavior is that users can modify columns whether or not they exist, and currently this requires allocation. Maybe this is a good opportunity to further distinguish `df[!, col]` from `df[:, col]` where

```julia
df[!, col_that_does_not_exist] .= value

```

will raise an error, and

```julia
df[:, col_that_does_exist] .= value

```

will not modify `df`

edit: ok after thinking about this more I am starting to understand the complexity. it is hard to make both operations consistent, especially if `df[:, c]` can be used as lvalue

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [October 27, 2022, 5:18pm UTC](https://discourse.julialang.org/t/what-if-df-col-v-was-in-place/89367/8 "2022-10-27T17:18:18Z")

</div>

The behavior of `df[:, :A]` was chosen for consistency with matrices. This can indeed give somewhat complex rules, probably if `getindex` returned views things would have been simpler for data frames.

BTW, let me stress that the DataFrames.jl behavior won’t change in a breaking way before it reaches 2.0, which isn’t currently planned and hopefully won’t have to happen soon as stability is essential for users.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [October 28, 2022, 1:26am UTC](https://discourse.julialang.org/t/what-if-df-col-v-was-in-place/89367/9 "2022-10-28T01:26:15Z")

</div>

> [@rafael.guerra](#):
>
> Shouldn’t the deviations from Julia’s base syntax be decorated with macros?

I agree with this. If you have this dataframe,

```julia
df = DataFrame(a=1:2)

```

then both of the following throw an `ArgumentError`:

```julia
df[!, :b]
df.b

```

Upon reading the Julia manual section on [customizing the broadcasting interface](https://docs.julialang.org/en/v1/manual/interfaces/#man-interfaces-broadcasting), one would conclude that `df[!, :b] .= 1` and `df.b .= 1` must throw an `ArgumentError`. There’s no way to get around the fact that `df[!, :b]` and `df.b` always error and thus do not return an object.

So, in order to get around this conundrum, DataFrames has overloaded two internal Julia functions: `Base.dotview` and `Base.dotgetproperty`. Unfortunately, these functions are internal and are not a part of any documented Julia interface. So attempting to reason about `df.b .= 1` based on prior Julia programming experience is a fruitless exercise. One just has to accept that DataFrames is special.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [October 28, 2022, 7:58am UTC](https://discourse.julialang.org/t/what-if-df-col-v-was-in-place/89367/10 "2022-10-28T07:58:36Z")

</div>

The more I work with tables and other datastructures, the more I prefer being explicit whether stuff like `x[:property] = ...`/`x.property = ...` inserts a new value or replaces an existing one. This often makes for more reliable and unambiguous code.

For tables, this approach seems a good fit for the `Accessors.jl` interface with its `@set`/`@insert` macros (and corresponding functions).  
An actual working example that shows consistency between getters/setters/inserters:

```julia
julia> using StructArrays, AccessorsExtra

julia> tbl = StructArray(a=1:3, b=[:x, :y, :z])
3-element StructArray(::UnitRange{Int64}, ::Vector{Symbol}) with eltype NamedTuple{(:a, :b), Tuple{Int64, Symbol}}:
 (a = 1, b = :x)
 (a = 2, b = :y)
 (a = 3, b = :z)

julia> tbl.b
3-element Vector{Symbol}:
 :x
 :y
 :z

julia> @delete tbl.b
3-element StructArray(::UnitRange{Int64}) with eltype NamedTuple{(:a,), Tuple{Int64}}:
 (a = 1,)
 (a = 2,)
 (a = 3,)

# @set: update existing column
julia> @set tbl.b = 1:3
3-element StructArray(::UnitRange{Int64}, ::UnitRange{Int64}) with eltype NamedTuple{(:a, :b), Tuple{Int64, Int64}}:
 (a = 1, b = 1)
 (a = 2, b = 2)
 (a = 3, b = 3)

# ... but not create a new one
julia> @set tbl.c = 1:3
ERROR: ArgumentError: Failed to assign properties (:c,) to object with properties (:a, :b).

# @insert: create a new column
julia> @insert tbl.c = 1:3
3-element StructArray(::UnitRange{Int64}, ::Vector{Symbol}, ::UnitRange{Int64}) with eltype NamedTuple{(:a, :b, :c), Tuple{Int64, Symbol, Int64}}:
 (a = 1, b = :x, c = 1)
 (a = 2, b = :y, c = 2)
 (a = 3, b = :z, c = 3)

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [October 28, 2022, 8:10am UTC](https://discourse.julialang.org/t/what-if-df-col-v-was-in-place/89367/11 "2022-10-28T08:10:02Z")

</div>

x-ref about the current design of indexing into a single column of a data frame: [DataFrames.jl indexing rules | Blog by Bogumił Kamiński](https://bkamins.github.io/julialang/2022/10/28/indexing.html).