# Release announcements for DataFrames.jl
**URL:** https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258
**Category:** Data
**Tags:** announcement, dataframes
**Created:** [December 3, 2018, 6:06pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258 "2018-12-03T18:06:13Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [December 5, 2018, 5:08pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/21 "2018-12-05T17:08:27Z")
Also there is this issue for fast path with checking if the data frame is sorted (actually what we would need is continuous blocks - not necessarily sorted, but this is harder to do), as this is a common case and also [https://github.com/h2oai/db-benchmark](https://github.com/h2oai/db-benchmark) has specific tests for it.
---
### Author:  [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)
#### Post date: [December 6, 2018, 12:19pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/22 "2018-12-06T12:19:49Z")
I have updated DataFrames package to 15 version. When I run the following command in juno/jupyter, it started throwing a warning like this.
Can someone explain this to me and the corrections to code?
```julia
using MLDataUtils #MLDataPattern,
@views begin
X,y=undersample((X_train,y_train),shuffle=false);
# X_resamp,y_resamp=oversample((X_train,y_train),fraction=0.2);
X = convert(Array{Float64}, X);
y = convert(Array{Int64}, y);
X_tst= convert(Array{Float64}, X_test);
y_tst = convert(Array{Int64}, y_test);
X_train0 = convert(Array{Float64}, X_train);
y_train0 = convert(Array{Int64}, y_train);
end
Warning: Indexing into a return value of columns on SubDataFrame will return a view of column value
│ caller = iterate at abstractarray.jl:838 [inlined]
└ @ Core .\abstractarray.jl:838
┌ Warning: Indexing into a return value of columns on SubDataFrame will return a view of column value
│ caller = iterate at abstractarray.jl:838 [inlined]
└ @ Core .\abstractarray.jl:838
```
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [December 6, 2018, 12:29pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/23 "2018-12-06T12:29:50Z")
You do not have to correct this. The warnings will go away in DataFrames 0.16.0 which should be released relatively soon.
The reason is that we want to introduce an optimization that when you access a column of a `SubDataFrame` it is not copied, but a `view` is returned which should be more efficient (especially for large data sets).
However, as this change is breaking; it will e.g. mean that when you mutate what you got you will mutate the source. Actually - this is what should have happend in the past (this is the contratct the DataFrames.jl promised, not to perform copy on column read) - but the implementation was different.
In 99% of use-cases this should not matter, so you can ignore the warnings.
---
### Author:  [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)
#### Post date: [December 6, 2018, 12:35pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/24 "2018-12-06T12:35:39Z")
Thanks. One more quesiton. using first(df,n) instead of head(df,6) gives this warning.
```julia
first(data,6)
Warning: In the future eachcol will have names argument set to false by default
│ caller = getmaxwidths(::DataFrame, ::UnitRange{Int64}, ::UnitRange{Int64}, ::Symbol) at show.jl:105
└ @ DataFrames C:\Users\chatura\.julia\packages\DataFrames\5Rg4Y\src\abstractdataframe\show.jl:105
```
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [December 6, 2018, 12:37pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/25 "2018-12-06T12:37:34Z")
This deprecation is already fixed in 0.15.1 as it was non-breaking, so please check-out latest release of the DataFrames.jl package.
---
### Author:  [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)
#### Post date: [December 6, 2018, 12:38pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/26 "2018-12-06T12:38:35Z")
Thanks for your quick responses.
---
### Author:  [@xiaodai](https://discourse.julialang.org/u/xiaodai)
#### Post date: [December 6, 2018, 1:40pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/27 "2018-12-06T13:40:32Z")
> [@bkamins](#):
>
> Also there is this issue for fast path with checking if the data frame is sorted
Alof of these ideas are already implemented in [FastGroupBy.jl](https://github.com/xiaodaigh/FastGroupBy.jl) I just need to update it for Julia v1 and we can start benchmarking! I think the InternedString.jl will yield HUGE improvements for string grouping, because once the strings are interned, we can group by performing a radix-sort on the pointers (which are just UInt64), which is super fast! In fact, I have created a faster radix-sort than data.table’s in [SortingLab.jl](https://github.com/xiaodaigh/SortingLab.jl); again I need to update it to Julia v1!
Can’t wait to release my updated benchmarks on Julia v1, but before that there is advent of code, work, set up personal website, disk.frame … I wish I can land a full time job to focus on the data ecosystem in Julia. Willing to take pay cut 🙂.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [December 9, 2018, 9:46pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/28 "2018-12-09T21:46:25Z")
I have updated [https://github.com/bkamins/Julia-DataFrames-Tutorial](https://github.com/bkamins/Julia-DataFrames-Tutorial) to DataFrames 0.15.2. You will see a bunch of deprecation warnings there (not that much but still a bit distirbing).
They will go away in DataFrames 0.16, but I released it now to highlight the upcoming changes in the underlying mechanics (mainly how `getindex`, `veiw` and `eachcol` work).
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [January 3, 2019, 9:00am UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/29 "2019-01-03T09:00:18Z")
DataFrames.jl 0.16 has been released. You can find the list of changes in the release notes here [Release Version 0.16.0 · JuliaData/DataFrames.jl · GitHub](https://github.com/JuliaData/DataFrames.jl/releases/tag/v0.16.0). The key change is finishing deprecation period for `getindex` and `view` methods.
The tutorial at [https://github.com/bkamins/Julia-DataFrames-Tutorial](https://github.com/bkamins/Julia-DataFrames-Tutorial) has been updated.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [January 18, 2019, 10:49pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/30 "2019-01-18T22:49:52Z")
DataFrames.jl 0.17 has been released (and if you are interested in these updated please subscribe to this thread for the future).
The [release notes](https://github.com/JuliaData/DataFrames.jl/releases) give the details of the changes. The most important changes are:
- Improved performance of split-apply-combine functions (👍 for @nalimilan).
- `view` is now fully functional and fast in all cases (we always remember the parent `DataFrame` without creating copies).
- Improved showing of data frames related types.
- Added `push!` for `DataFrameRow`.
- `DataFrame` constructor accepts `SubDataFrame` and `DataFrameRow` objects.
My tutorial [https://github.com/bkamins/Julia-DataFrames-Tutorial](https://github.com/bkamins/Julia-DataFrames-Tutorial) has been updated to include the new functionality.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [January 19, 2019, 1:05pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/31 "2019-01-19T13:05:55Z")
A small, but relevant update. With this version of DataFrames.jl we also re-export a new version of Missings.jl, which introduces `passmissing` function. We would welcome a feedback on this functionality (as it might eventually be considered for inclusion into Julia Base).
The `passmissing` function wraps any Julia function `f` (typically `missing` “unaware”) accepting positional arguments so that if any of them is `missing` it returns `missing` and otherwise calls `f` with the passed arguments. Here is an example showing you the difference:
```julia
julia> string(missing, " ", missing)
"missing missing"
julia> passmissing(string)(missing, " ", missing)
missing
```
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [February 5, 2019, 12:30pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/32 "2019-02-05T12:30:23Z")
We have released DataFrames.jl version 0.17.1. [Here](https://github.com/JuliaData/DataFrames.jl/releases/tag/v0.17.1) you can read release notes.
Apart from polishing corner cases, the thing that a typical user might be interested in is that two convenience functions for working with `GroupedDataFrame` were introduced:
- `groupvars`: gives you a vector of names of columns in the parent data frame used for grouping
- `groupindices`: gives you a vector of row group indices in the parent data frame
[https://github.com/bkamins/Julia-DataFrames-Tutorial](https://github.com/bkamins/Julia-DataFrames-Tutorial) has been updated to reflect the changes in the release.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [April 26, 2019, 3:17pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/33 "2019-04-26T15:17:47Z")
We have released DataFrames.jl version 0.18.0. [Here](https://github.com/JuliaData/DataFrames.jl/releases/tag/v0.18.0) you can read release notes.
This is a big release. The main highlights are:
- functions that create a new `DataFrame` copy passed columns by default; this can be overridden by copycols keyword argument or by using the DataFrame! function that does not copy passed columns; this means that we take **safety first** approach and you have to **opt in for speed** when creating a new `DataFrame`; we believe that this is a good choice as even experienced users of the package were getting caught by the old behavior (and it is especially targeted at making the usage less error prone for entry-level users);
- in order to simplify the work with data frames after the change of default `DataFrame` construction there were significant improvements in `append!`, `push!` and `vcat` methods to make the workflow using them more smooth;
- following requests we have thinned the dependency list of DataFrames.jl which should improve its loading times (the change is mildly breaking so if anyone experiences serious problems because of this please report an issue in DataFrames.jl);
- there is a long list of finished deprecation periods for functions and new deprecation messages; the reason is that we want to clean up the package before 1.0 release (also the list of deprecated features got quite long over the years, we will continue with this process also in the coming releases with the objective to have no deprecated functionalities in 1.0 release);
- there are significant improvements in various `show` method implementations
- optimized methods for `PooledArrays` in split-apply-combine were added
I will update Julia-DataFrames-Tutorial soon (and add a link here when it is updated).
In the coming release the next big thing (hopefully) will be making `setindex!` and broadcasting behavior of data frames consistent with Julia 1.0 standards.
## Update
I have updated the tutorial [here](https://github.com/bkamins/Julia-DataFrames-Tutorial).
There were some minor rough edges reported ([here](https://github.com/JuliaData/DataFrames.jl/pull/1789) and [here](https://github.com/JuliaData/DataFrames.jl/pull/1790)) so we will soon make a patch release.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [April 28, 2019, 10:08pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/34 "2019-04-28T22:08:26Z")
A patch release 0.18.1 of DataFrames.jl has just been registered. The release makes `DataFrame` constructor behave correctly when passed some objects that met Tables.jl interface (most importantly database connections). Thanks for @cadoubs and @quinnj for helping with the issue.
---
### Author:  [@cadoubs](https://discourse.julialang.org/u/cadoubs)
#### Post date: [April 29, 2019, 7:26am UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/35 "2019-04-29T07:26:35Z")
Thank you for your fast analysis and patch. I validate the update on my application with MySQL.jl
You make a very nice job.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [April 29, 2019, 1:39pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/36 "2019-04-29T13:39:09Z")
I have just stumbled on [an old question](https://stackoverflow.com/questions/46143997/reading-json-array-into-julia-dataframe-like-type) on StackOverflow that lacked a good answer for over two years. It is related to a conversion from JSON to a columnar data structure (looking at the number of stars on the question I guess it is a pattern that is needed quite often).
Thanks to the new `vcat` functionality proposed by [pdeffebach](https://github.com/pdeffebach) and @oxinabox in DataFrames 0.18 we can do such things cleanly.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [May 3, 2019, 11:17am UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/37 "2019-05-03T11:17:18Z")
DataFrames.jl v0.18.2 patch release was just merged. [Here](https://github.com/JuliaData/DataFrames.jl/releases/tag/v0.18.2) you can read about the changes it introduces.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [July 15, 2019, 5:40pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/38 "2019-07-15T17:40:19Z")
DataFrames.jl v0.19.0 has just been released. It is a major release towards DataFrames.jl 1.0 (we cannot get there yet as we have to go through deprecation cycle).
The number of changes is significant and includes:
**API changes:**
- allow `Regex` indexing of columns
- allow `Not` from [InvertedIndices.jl](https://github.com/mbauman/InvertedIndices.jl) indexing of rows and columns
- add `!` indexing of rows of `AbstractDataFrame`
- deprecate indexing with column or columns only (like `df[:a]` or `df[1:2]` )
- define target rules for `getindex` , `getproperty, ` setindex! `, and` setproperty! `for` AbstractDataFrame `and` DataFrameRow` (in this release old behavior is deprecated; in the next release wit will get replaced by target functionality)
- add indexing using `CartesianIndex{2}` for `AbstractDataFrame`
- full support of broadcasting for `AbstractDataFrame`
- support for broadcasting assignment for `DataFrameRow`
- `keys(::DataFrameRow)` now returns a `Tuple` of column names
- added `get` and `map` methods for `DataFrameRow`
- `categorical!` now accepts columns that contain `missing` values
- `get` and `haskey` for `AbstractDataFrame` is deprecated now
- `empty!` for `DataFrame` is deprecated now
- add `hasproperty` for `AbstractDataFrame`
**Fixes:**
- improved showind `DataFrameRow` with zero columns
- fix `combine` with aggregation when `skipmissing=true`
**Minor changes:**
- improvements in error messages and types of thrown exceptions on error
- various documentation improvements
- improved `getindex` speed for vector of `Bool` indexing
- remove InteractiveUtils.jl dependency
The major change is change of indexing rules and full support for broadcasting. [Here](https://juliadata.github.io/DataFrames.jl/stable/lib/indexing.html) are the details. In general in the design there was a tension between: ease of use, flexibility, safety and consistency.
Here are the major highlights:
- you can use `Not` and `Regex` for column indexing
- `df[col]` is now `df[!, col]` and gets/replaces a column in a data frame “as is”
- `df[:, col]` will always get a copy of a column/set a column in place
- `df[cols]` is now `df[!, cols]` and gets a new data frame without copying of columns
- `df[:, cols]` and gets a new data frame with copying of columns
- `df.col` is the same as `df[!, col]` for consistency with Base indicating that it gives you “as is” access to the property of the data frame (i.e. it gives you the column without copying and replaces the column)
- data frames can take part in broadcasting
- You can perform broadcasting assignment to `AbstractDataFrame` and `DataFrameRow`; as a special rule: using `df[!, col]` syntax you can create a new column/replace old one using broadcasting (something which is non standard in regular broadcasting which is always in-place).
In summary `!` indicates “an unsafe” operation. The reason is that people often were tricked by getting columns of a data frame, mutating them (e.g. resizing or sorting), and in consequence corrupting the source data frame. Now we hope that `!` will serve them as a warning that this is not a safe operation (as opposed to `:` indexing which always makes a copy).
Here are the new rules at work:
```julia
julia> df = DataFrame(x1=1:3, x2=2:4, y='a':'c')
3×3 DataFrame
│ Row │ x1 │ x2 │ y │
│ │ Int64 │ Int64 │ Char │
├─────┼───────┼───────┼──────┤
│ 1 │ 1 │ 2 │ 'a' │
│ 2 │ 2 │ 3 │ 'b' │
│ 3 │ 3 │ 4 │ 'c' │
julia> select(df, r"x")
3×2 DataFrame
│ Row │ x1 │ x2 │
│ │ Int64 │ Int64 │
├─────┼───────┼───────┤
│ 1 │ 1 │ 2 │
│ 2 │ 2 │ 3 │
│ 3 │ 3 │ 4 │
julia> select(df, Not(r"x"))
3×1 DataFrame
│ Row │ y │
│ │ Char │
├─────┼──────┤
│ 1 │ 'a' │
│ 2 │ 'b' │
│ 3 │ 'c' │
julia> df[Not(1), Not(1)]
2×2 DataFrame
│ Row │ x2 │ y │
│ │ Int64 │ Char │
├─────┼───────┼──────┤
│ 1 │ 3 │ 'b' │
│ 2 │ 4 │ 'c' │
julia> df .+ 1
3×3 DataFrame
│ Row │ x1 │ x2 │ y │
│ │ Int64 │ Int64 │ Char │
├─────┼───────┼───────┼──────┤
│ 1 │ 2 │ 3 │ 'b' │
│ 2 │ 3 │ 4 │ 'c' │
│ 3 │ 4 │ 5 │ 'd' │
julia> df .+= ones(Int, size(df))
3×3 DataFrame
│ Row │ x1 │ x2 │ y │
│ │ Int64 │ Int64 │ Char │
├─────┼───────┼───────┼──────┤
│ 1 │ 2 │ 3 │ 'b' │
│ 2 │ 3 │ 4 │ 'c' │
│ 3 │ 4 │ 5 │ 'd' │
julia> df[!, :z] .= 1
3-element Array{Int64,1}:
1
1
1
julia> df
3×4 DataFrame
│ Row │ x1 │ x2 │ y │ z │
│ │ Int64 │ Int64 │ Char │ Int64 │
├─────┼───────┼───────┼──────┼───────┤
│ 1 │ 2 │ 3 │ 'b' │ 1 │
│ 2 │ 3 │ 4 │ 'c' │ 1 │
│ 3 │ 4 │ 5 │ 'd' │ 1 │
```
---
### Author:  [@pazzo83](https://discourse.julialang.org/u/pazzo83)
#### Post date: [July 15, 2019, 5:47pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/39 "2019-07-15T17:47:46Z")
this is great great stuff! Is there anywhere where you guys have listed the goals for 1.0, or roughly what is going to be materially different from the structure of a DataFrame now?
---
### Author:  [@oxinabox](https://discourse.julialang.org/u/oxinabox)
#### Post date: [July 15, 2019, 6:03pm UTC](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258/40 "2019-07-15T18:03:32Z")
[https://github.com/JuliaData/DataFrames.jl/issues/1678](https://github.com/JuliaData/DataFrames.jl/issues/1678)
[Previous page](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258.md?page=1)
[Next page](https://discourse.julialang.org/t/release-announcements-for-dataframes-jl/18258.md?page=3)