# Release announcements for DataFrames.jl

**URL:** <https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258>\
**Category:** Data\
**Tags:** announcement, dataframes\
**Created:** [December 3, 2018, 6:06pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258 "2018-12-03T18:06:13Z")\
**Posts on this page:** 20\
**Page:** 4

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [July 19, 2019, 6:47am UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/61 "2019-07-19T06:47:28Z")

</div>

Regarding `view` vs `!` the difference is the following:

`!` allows you to **mutate** the bindings in internal structure of `df`. Therefore when you write:

```julia
df[!, :col] = vector

```

you can create a column if it does not exist and replace it if it exists. The same is with broadcasting assignment like:

```julia
df[!, :col] .= x

```

using a `view` would not allow this.

Now for `getindex` side, there is an ongoing difference what we should do (see [Make `getproperty(df, col)` return a full length view of the column · Issue #1844 · JuliaData/DataFrames.jl · GitHub](https://github.com/JuliaData/DataFrames.jl/issues/1844#issuecomment-512292427)). First, the current major difference is speed (this probably could be improved):

```julia
julia> @benchmark df[!, 1]
BenchmarkTools.Trial:
  memory estimate: 0 bytes
  allocs estimate: 0
  --------------
  minimum time: 15.746 ns (0.00% GC)
  median time: 15.948 ns (0.00% GC)
  mean time: 18.275 ns (0.00% GC)
  maximum time: 157.673 ns (0.00% GC)
  --------------
  samples: 10000
  evals/sample: 997

julia> @benchmark @view df[:, 1]
BenchmarkTools.Trial:
  memory estimate: 48 bytes
  allocs estimate: 1
  --------------
  minimum time: 308.907 ns (0.00% GC)
  median time: 312.146 ns (0.00% GC)
  mean time: 358.267 ns (0.66% GC)
  maximum time: 5.262 μs (93.72% GC)
  --------------
  samples: 10000
  evals/sample: 247

```

then the `view` and the raw-access to a vector is not always the same thing type-wise (it may affect method dispatch downstream, it does for example for `CategoricalArray`).

Finally - people already complained that `df[!, :col]` is longer to write than `df[:col]` and `@view df[:, :col]` would be even longer to write.

In summary - we needed `!` for “write” side of the `getindex`/`setindex!`/broadcasting assignment combo anyway. Then we had do define what it does for “read” side to be consistent. As of today we have decided that it will do the same what `df[:col]` did (as removing an equivalent of `df[:col]` for reading data from a data frame would: hurt performance and lead to code breakage).

Note that with what we have implemented you simply add `!, ` to your code and you know it will work as it used to without having to thing about it; if we switched to `@view` approach - which was considered - a lot of code would be broken; Actually the deprecation of `df[:col]` would be `parent(@view df[:, col])` but we considered that this would not be acceptable. With a package like DataFrames.jl we had to consider the fact that people have 5+ years of accumulated code using it and if we were going to be breaking (which we decided to do) the “fixing” should be easy and not noisy visually.

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [July 19, 2019, 10:16am UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/62 "2019-07-19T10:16:04Z")

</div>

What about df[rows, :] vs df[rows, !] ?  
Is it the same story for rows?

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [July 19, 2019, 11:21am UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/63 "2019-07-19T11:21:20Z")

</div>

No - `!` is only needed for row selector.  
The reason is that internally `DataFrame` is a collection of columns (not rows).

For column selector you can use:

- a `Symbol`
- an `Integer` other than `Bool`
- a vector of `Integers` other than `Bool` (also with abstract `eltype`)
- a vector of `Bool`
- a vector of `Symbol`
- a `Colon`
- a `Regex`
- a `Not` expression accepting any of the above as the argument

(which I hope is flexible enough to cover all use-cases in a user-friendly way)

---

<div class="post-metadata">

**Author:** ![Christopher\_Fisher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christopher_fisher/32/26132_2.png) [@Christopher\_Fisher](https://discourse.julialang.org/u/Christopher_Fisher)\
**Post date:** [July 19, 2019, 2:44pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/64 "2019-07-19T14:44:49Z")

</div>

Overall, I think the changes are great. One thing I miss, however, is the ability to assign a scalar to a new column. The following no longer works:

```julia
using DataFrames
df = DataFrame()
v = .3
df[!,:a] = v

ERROR: MethodError: no method matching setindex!(::DataFrame, ::Float64, ::typeof(!), ::Symbol)
Closest candidates are:
  setindex!(::DataFrame, ::AbstractArray{T,1} where T, ::typeof(!), ::Union{Signed, Symbol, Unsigned}) at /home/dfish/.julia/packages/DataFrames/GoFnP/src/dataframe/dataframe.jl:465
  setindex!(::DataFrame, ::Any, ::Colon, ::Any) at /home/dfish/.julia/packages/DataFrames/GoFnP/src/deprecated.jl:1595
  setindex!(::DataFrame, ::Any, ::Integer, ::Union{Signed, Symbol, Unsigned}) at /home/dfish/.julia/packages/DataFrames/GoFnP/src/dataframe/dataframe.jl:474
  ...
Stacktrace:
 [1] top-level scope at none:0

```

The solution is to wrap v in an array like so: `[v]`. The downside is that there is no elegant way to use the same code when v could be a scalar or vector. Is there a reason that this is no longer possible?

---

<div class="post-metadata">

**Author:** ![aaowens](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aaowens/32/12101_2.png) [@aaowens](https://discourse.julialang.org/u/aaowens)\
**Post date:** [July 19, 2019, 3:00pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/65 "2019-07-19T15:00:25Z")

</div>

You can broadcast to construct a column of the scalar once the DataFrame has something in it.

```julia
julia> v = .3
julia> df[!, :b] = rand(100);
julia> df[!, :a] .= v
100-element Array{Float64,1}:
 0.3
 0.3
 0.3
 0.3
 0.3
 0.3
 0.3
 0.3
 0.3
 0.3
 ⋮  
 0.3
 0.3
 0.3
 0.3
 0.3
 0.3
 0.3
 0.3
 0.3

```

Or did you actually want the DataFrame to hold a single scalar, instead of an array?

---

<div class="post-metadata">

**Author:** ![Christopher\_Fisher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christopher_fisher/32/26132_2.png) [@Christopher\_Fisher](https://discourse.julialang.org/u/Christopher_Fisher)\
**Post date:** [July 19, 2019, 3:10pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/66 "2019-07-19T15:10:05Z")

</div>

Yeah, the latter. In previous versions, it would promote a scalar to an array and create a new column, assuming the dataframe is empty or has only one row. Maybe DataFrames functions more consistently without this behavior.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [July 19, 2019, 3:44pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/67 "2019-07-19T15:44:43Z")

</div>

There is already a PR for an empty data frame case here [https://github.com/JuliaData/DataFrames.jl/pull/1890](https://github.com/JuliaData/DataFrames.jl/pull/1890).

In consequence writing:

```julia
df = DataFrame()
df[!, :col] .= 1

```

will be possible but will create a `0`-element vector. The reasoning behind it is that `df` has `0`-rows so adding a column to it should keep it this way.

Do you think a data frame with zero columns should have a different behavior than an empty data frame with at least one column here?

Also - is this case really a significant one? (I am aware that it might be popular when learning the package but does it happen in real cases).

Also note that you can do the following `push!(df, (a=1,))`, where `df` was created using `DataFrame()`.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [July 19, 2019, 4:14pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/68 "2019-07-19T16:14:12Z")

</div>

Just for a reference. You can assign to a new column in general like:

```julia
df = DataFrame(a=[1,2,3])
df[!, :b] .= 1

```

works OK. The only problematic cases were zero column data frame and an empty data frame with more than zero columns (they are currently handled in the same way in the PR I have mentioned).

---

<div class="post-metadata">

**Author:** ![Christopher\_Fisher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christopher_fisher/32/26132_2.png) [@Christopher\_Fisher](https://discourse.julialang.org/u/Christopher_Fisher)\
**Post date:** [July 19, 2019, 4:33pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/69 "2019-07-19T16:33:43Z")

</div>

Thanks for the explanation. In your example, my naive expectation would be that `df[!,:col] .= 1` creates a column of `Int64[]` called col with a single element of 1:

```julia
 df
  1×1 DataFrame
│ Row │ a │
│ │ Int64 │
├─────┼───────┤
│ 1 │ 1 │

```

It is somewhat of an edge case. Although I would expect it to create a new column and assign 1, that may not be consistent with the current behavior, where it would create a column of size R where R is the number of rows. For example:

> using DataFrames  
> df = DataFrame()  
> R = 10  
> df[!,:a] = rand( R)  
> df[!,:b] .= .4

```julia
df
  10×2 DataFrame
│ Row │ a │ b │
│ │ Float64 │ Float64 │
├─────┼──────────┼─────────┤
│ 1 │ 0.859254 │ 0.4 │
│ 2 │ 0.931467 │ 0.4 │
│ 3 │ 0.276189 │ 0.4 │
│ 4 │ 0.695778 │ 0.4 │
│ 5 │ 0.436747 │ 0.4 │
│ 6 │ 0.478671 │ 0.4 │
│ 7 │ 0.220407 │ 0.4 │
│ 8 │ 0.778795 │ 0.4 │
│ 9 │ 0.614729 │ 0.4 │
│ 10 │ 0.825508 │ 0.4 │

```

I’m not entirely certain what should be done when R = 0 or R is undefined because the dataframe is empty On one hand, it seems odd to create an empty column when a specific value is being supplied, but assigning the value does not seem entirely consistent with the behavior above either. I guess, between the two options, I might choose to have it create the column and assign the value and use something like `df[!,:a] .= Int64[]` to create an empty column. Of course, that is my somewhat under-informed assessment, and I have not thought about all of the ramifications.

The current behavior is slightly inconvenient because I have to wrap scalars in an array, at least when creating the initial column, e.g. `df = DataFrame(); df[!,:a] = [.3]`. After that, as you pointed out in your second comment to me, it is possible to assign values subsequently. That’s not a big deal if I always know where the first column will be created. What is more problematic is if `v` could be a scalar _or_ vector, and the DataFrame is empty. As far as I know, this would require two sets of syntax to handle both cases, whereas before, it handled the assignment of both scalars and vectors.

I agree with you. I’m not sure how often this problem arises in actual code. I have encountered it from time to time in my work. This occurs when I need to extract data from various sources and integrate it with some container iteratively and flexibly (i.e. I may not know what the columns will be or what their sizes will ultimately be).

---

<div class="post-metadata">

**Author:** ![Christopher\_Fisher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christopher_fisher/32/26132_2.png) [@Christopher\_Fisher](https://discourse.julialang.org/u/Christopher_Fisher)\
**Post date:** [July 19, 2019, 4:47pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/70 "2019-07-19T16:47:13Z")

</div>

P.S. I couldn’t adapt your example with push to my problem:

Case 1: scalar

```julia
df = DataFrame()
a = 1
push!(df,(a=a,))

```

```julia
 df
  1×1 DataFrame
│ Row │ a │
│ │ Int64 │
├─────┼───────┤
│ 1 │ 1 │

```

Case 2: vector

```julia
df = DataFrame()
a= [1,2]
push!(df,(a=a,))

```

```julia
 df
  1×1 DataFrame
│ Row │ a │
│ │ Array… │
├─────┼────────┤
│ 1 │ [1, 2] │

```

I tried various placements of the `"..."` operator, but it would not accommodate both cases.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [July 19, 2019, 4:48pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/71 "2019-07-19T16:48:01Z")

</div>

We could treat `DataFrame()` as having **undefined** number of rows (as opposed to `0` rows).  
Thinking about it we then should also allow:

```julia
df = DataFrame()
df[!, :c] .= [1,2,3]

```

(and create a 3 element vector - now it throws an error; I will update the PR I have mentioned above and we can discuss it further there)

PS. `push!` is only for scalars

---

<div class="post-metadata">

**Author:** ![Christopher\_Fisher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christopher_fisher/32/26132_2.png) [@Christopher\_Fisher](https://discourse.julialang.org/u/Christopher_Fisher)\
**Post date:** [July 19, 2019, 4:56pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/72 "2019-07-19T16:56:04Z")

</div>

Thanks for your time. I found the discussion helpful for understanding DataFrames.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [July 19, 2019, 8:13pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/73 "2019-07-19T20:13:53Z")

</div>

I have implemented the requested changes in [https://github.com/JuliaData/DataFrames.jl/pull/1890](https://github.com/JuliaData/DataFrames.jl/pull/1890). The functionality is tricky so if someone can have a look at it it would be appreciated.

---

<div class="post-metadata">

**Author:** ![Christopher\_Fisher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christopher_fisher/32/26132_2.png) [@Christopher\_Fisher](https://discourse.julialang.org/u/Christopher_Fisher)\
**Post date:** [July 19, 2019, 8:44pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/74 "2019-07-19T20:44:27Z")

</div>

Thank you. I will take a look.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [July 26, 2019, 2:24pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/75 "2019-07-26T14:24:23Z")

</div>

Based on the feedback after 0.19.0 release we have a patch 0.19.1 release that introduces the following changes:

- we drop StatsBase.jl dependency, which improves DataFrames.jl load time
- `push!` and `append!` now make sure they do not produce an output data frame with unequal number of rows in columns (this was a common problem causing hard-to-catch bugs)
- `join`, `groupby` and `show`-related functions now check if the data frame they work on is not internally corrupted (most often the problem is unequal number of rows in columns); this allows users to track possible column aliasing related bugs in their code easier
- broadcasting now allows broadcasting into `0`-row (empty) data frame and correctly handles broadcasting into a single cell of a data frame.

Thanks for all that reported problems and contributed!

You can see the detailed release notes [here](https://github.com/JuliaData/DataFrames.jl/releases/tag/v0.19.1).

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [August 7, 2019, 5:40pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/76 "2019-08-07T17:40:23Z")

</div>

We have just released a 0.19.2 patch release. The full log of changes can be found [here](https://github.com/JuliaData/DataFrames.jl/releases/tag/v0.19.2).

The most relevant changes are:

- `disallowmissing` , `allowmissing` and `categorical` functions were added
- `unstack` now accepts `renamecols` keyword argument providing a flexible mechanizm to generate column names of an unstacked data frame

---

<div class="post-metadata">

**Author:** ![sylvaticus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaticus/32/203883_2.png) [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Post date:** [August 9, 2019, 1:59pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/77 "2019-08-09T13:59:28Z")

</div>

I found some weird syntax in DataFrame 0.19.2… is it me doing wrong (i.e. “is there a better way to do it”) or it is supposed to be like this ?

Let’s assume the following df:

```julia
df = DataFrame(a=[1,2,3],b=[4,5,6],c=[7,8,9])

# Change column order
df = df[[:b, :a, :c]] # before
df = df[:,[:b, :a, :c]] # now

# Operate on a column whose name is memorised in a variable
C = :c; eltype(df[C]) # before 
C = :c; eltype(df[!,C]) # now

# Delete cols
deletecols!(df,:b) # before
select!(df,Not(:b)) # now

```

What the new `!` symbol stand for ? How `df[!,:a]` is different than `df[:,:a]` ?

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [August 9, 2019, 2:21pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/78 "2019-08-09T14:21:52Z")

</div>

`!` roughly stands for “a direct access to a column stored in a data frame”, as opposed to `:` which stands for “access to a copy of a column stored in a data frame”.

The detailed rules are laid out [here](https://juliadata.github.io/DataFrames.jl/stable/lib/indexing/).

The reason for this change is described above in this thread (around post 51), but in short a direct access to a column lead to many nasty bugs when using DataFrames.jl package (people accessed the column, then mutated it and they were not aware that they also mutated the contents of the source data frame). The `!` symbol adds a bit of verbosity, but is meant to warn that the operation is _potentially_ mutating (so this similar with the meaning of `!` in Base in function names).

Note that instead of `df[!, :col]` you can just write `df.col`.

Also note that:

- to change the order of columns you can use the `permutecols!` function which works in place; if you want to avoid copying of columns and create a new data frame use `df[!, [:b, :a, :c]]` (this is potentially unsafe though - as explained above)
- `deletecols!` was deprecated because `select!` just does the job with `Not` indexing, so there would be a duplication of functionality (and `select`-family of functions is something that many people already know from other data processing ecosystems).

---

<div class="post-metadata">

**Author:** ![dmolina](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmolina/32/5246_2.png) [@dmolina](https://discourse.julialang.org/u/dmolina)\
**Post date:** [August 9, 2019, 2:30pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/79 "2019-08-09T14:30:12Z")

</div>

I write to say that in my opinion the new syntax is nice. The symbol ! inside the selection is a bit strange at first, but after you got use it has a lot of sense. The other changes are also good in mi opinion.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [August 16, 2019, 6:43am UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/80 "2019-08-16T06:43:13Z")

</div>

I have a small announcement. [DataFrames Tutorial](https://github.com/bkamins/Julia-DataFrames-Tutorial) is for some time in sync with DataFrames.jl 0.19 release. But I have just added in the 04\_loadsave.ipynb notebook examples of integration with JSONTables.jl.

@quinnj has done a fantastic job with JSONTables.jl and with the release 0.1.2 of this package you can safely read and write JSON data (both row- and column- oriented) to a `DataFrame`.

[Previous page](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258.md?page=3)

[Next page](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258.md?page=5)