Indexing DataFrame with : does not generate a copy

Since I’ve tripped over the same issue, I’ll illustrate below and then paste what @bkamins patiently explained in a Slack forum. This involves getindex and setindex.

using DataFrames


df = DataFrame(A=1:3, B=4:6)

### getindex (: used in df on the right hand side of =)
copyOfA = df[:, :A]

copyOfA .= 0

copyOfA

# note that :A has not changed
df


### setindex (: used in df on left hand side of =)

df[:, :A] .= 0

# now df has changed
df

From Slack discussion:

The issue is that you are reading the documentation of getindex,
while we were discussing the behavior of setindex!/broadcasting assignment context.

The design principle in DataFrames.jl is to provide every option
the user might ask for (as it is a low-level package.
High level packages like DataFramesMeta.jl or DataFrameMacros.jl
expose less to the user but under a simpler API).

In getindex context a user can ask for the following behaviors:
get a vector “as is”, without copying; this is achieved with df[!, :a] and df.a;
get a vector with copying; that is achieved with df[:, :a];
get a view of the vector; this is achieved with view(df, :, :a) or view(df, !, :a)

In setindex!/broadcasting assignment context
the user might ask for the following:

set an existing vector in-place,
 this is done by df[:, :a] = vec and df[:, :a] .= something

replace an existing vector without copying:
 df[!, :a] = vec or df.a = vec

replace an existing vector with copying:
 df[!, :a] .= vec and df[!, :a] .= something

add a new vector without copying
 df[!, :new] = vec or df.new = vec

add a new vector with copying
df[:, :new] = vec or df[:, :new] .= something
or 
df[!, :new] .= something

(this last rule breaks things a bit as ! and : behave here
the in same way, but it was added for user convenience)

These are selected basic rules. All the rules of indexing
are described in Indexing · DataFrames.jl.

In short: the range of behaviors users might want is very vast,
so we needed to introduce both : and ! to cover every option.

3 Likes