# Announcement: An Update on DataFrames Future Plans

**URL:** <https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646>\
**Category:** Data\
**Tags:** announcement\
**Created:** [July 4, 2017, 5:50pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646 "2017-07-04T17:50:56Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [July 7, 2017, 1:00pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/21 "2017-07-07T13:00:56Z")

</div>

There are codegen Union optimizations that have already been merged to do the call-site method splitting @malmaud mentioned above; unfortunately, there have been issues inlining after the method splitting, so the optimizations aren’t fully realized (I believe this is fully fixed in master though). The additional optimizations being worked on in Jameson & my PRs are related to allowing `Union{T, S, ...}` to be “unboxed” when `T` and `S` (and `...`) are all `isbits` types. This affects struct fields and array elements, allowing more efficient memory layout and access/setting.

---

<div class="post-metadata">

**Author:** ![malmaud](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/malmaud/32/29_2.png) [@malmaud](https://discourse.julialang.org/u/malmaud)\
**Post date:** [July 7, 2017, 3:23pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/24 "2017-07-07T15:23:27Z")

</div>

Oh great. Might you have any example of using `@code_llvm` or what have you that shows call-side method splitting in action?

---

<div class="post-metadata">

**Author:** ![malmaud](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/malmaud/32/29_2.png) [@malmaud](https://discourse.julialang.org/u/malmaud)\
**Post date:** [July 7, 2017, 3:26pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/25 "2017-07-07T15:26:10Z")

</div>

NM, I think I have one:

```julia
f(x::Int)=x+1
f(::Void) =nothing
g(x) = x>0 ? f(x) : f(nothing)
@code_llvm g(0)

```

```julia
define { i8**, i8 } @julia_g_61178([8 x i8]* noalias nocapture, i64) #0 !dbg !5 {
top:
  %2 = icmp slt i64 %1, 1
  br i1 %2, label %L3, label %if

if: ; preds = %top
  %3 = add i64 %1, 1
  %4 = bitcast [8 x i8]* %0 to i8**
  %5 = insertvalue { i8 **, i8 } undef, i8** %4, 0
  %6 = insertvalue { i8**, i8 } %5, i8 1, 1
  %7 = bitcast [8 x i8]* %0 to i64*
  store i64 %3, i64* %7, align 1
  br label %L3

L3: ; preds = %if, %top
  %merge = phi { i8**, i8 } [{ i8** null, i8 2 }, %top], [%6, %if]
  ret { i8**, i8 } %merge
}

```

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 7, 2017, 4:57pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/26 "2017-09-07T16:57:26Z")

</div>

@nalimilan, is there an update on this? I’ve noticed that all of the `DataTables` commits have been merged with `DataFrames` and that `DataFrames` now uses `Nulls` instead of `DataArrays`. This is definitely a welcome update, I’d like to start migrating my packages from `DataTables` to `DataFrames` in anticipation of whatever the world will look like in 1.0, so I wanted to try to get some idea of how stable current plans are.

Also, @quinnj has overhauled `DataStreams`, and I’d like to get things up and running with the new version, I was wondering if there was any willingness to start porting things over (I can definitely help with this).

Thanks to everyone for their ongoing efforts.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [September 8, 2017, 9:41am UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/27 "2017-09-08T09:41:19Z")

</div>

Yes, I wanted to post an short update but you beat me to it. You summarized the situation pretty well. Users and package authors are encouraged to test the master branch of DataFrames, which is now based on Nulls.jl and reflects the new API. The branch should be fully usable, but it will not be as fast on Julia 0.6 as on 0.7. Please report any issues on GitHub. Benchmarks on Julia 0.7 (comparing performance with the DataArrays-based DataFrames) would be particularly useful.

The main limitation right now is that most packages depending on DataFrames have no yet been updated (e.g. Query and RCall; there’s an [open pull request](https://github.com/JuliaStats/StatsModels.jl/pull/30) for StatsModels). But DataStreams, Feather and CSV have already been ported. Of course new releases of dependent packages cannot be tagged before the new DataFrames itself has been tagged, so until we’re ready to make a release that code will have to live in package branches.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 8, 2017, 3:27pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/28 "2017-09-08T15:27:06Z")

</div>

Great. Do we know yet whether the promised improvements to union types have already been implemented in 0.7? I haven’t seen anything about it in NEWS.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [September 8, 2017, 4:28pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/29 "2017-09-08T16:28:40Z")

</div>

Yes, the Union struct/array optimizations have landed in Base. There is still an [open issue](https://github.com/JuliaLang/julia/issues/23338) to improve inference (currently planned and in progress), as well as a SIMD isbits Union array [issue](https://github.com/JuliaLang/julia/issues/23336) that will hopefully be helped by the first issue and some other planned codegen cleanups. Both are on track for 1.0.

I’ve been letting the dust settle a bit on the isbits Union optimizations before doing a NEWS entry (we’ve had a couple bug reports come in that have all been fixed). I’m also planning a more details blog post on the optimizations once all the issues have been cleaned up/finished.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [September 8, 2017, 11:44pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/30 "2017-09-08T23:44:22Z")

</div>

> [@nalimilan](#):
>
> The main limitation right now is that most packages depending on DataFrames have no yet been updated (e.g. Query and RCall; there’s an open pull request for StatsModels).

I just added this PR [Add support for Nulls.jl based DataFrames.jl by davidanthoff · Pull Request #60 · queryverse/IterableTables.jl · GitHub](https://github.com/davidanthoff/IterableTables.jl/pull/60) that should add support for the new [Nulls.jl](https://github.com/JuliaData/Nulls.jl) based [DataFrames.jl](https://github.com/JuliaStats/DataFrames.jl) to [IterableTables.jl](https://github.com/davidanthoff/IterableTables.jl) (and thereby to [Query.jl](https://github.com/davidanthoff/Query.jl) and all the other packages in that ecosystem).

Though note that [Query.jl](https://github.com/davidanthoff/Query.jl) and the rest of the ecosystem around it will not move to the `Union{T,Null}` approach, but will continue to use [DataValues.jl](https://github.com/davidanthoff/DataValues.jl) for representing missing values. The whole `Union{T,Null}` design has some fundamental issues that make it more or less unusable within [Query.jl](https://github.com/davidanthoff/Query.jl), as far as I can tell at this point (I’d be happy to be corrected on this point). So the situation we’ll have is that [Query.jl](https://github.com/davidanthoff/Query.jl) and the iterable tables universe will be able to interop with the [Nulls.jl](https://github.com/JuliaData/Nulls.jl) based [DataFrames.jl](https://github.com/JuliaStats/DataFrames.jl), but will use a different missing value story both under the hood and within any query users write.

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [September 9, 2017, 1:08am UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/31 "2017-09-09T01:08:41Z")

</div>

> [@davidanthoff](#):
>
> The whole Union{T,Null} design has some fundamental issues that make it more or less unusable within Query.jl

☹ Do you have a writeup explaining the issues?

---

<div class="post-metadata">

**Author:** ![gabrielgellner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gabrielgellner/32/75_2.png) [@gabrielgellner](https://discourse.julialang.org/u/gabrielgellner)\
**Post date:** [September 9, 2017, 1:53am UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/32 "2017-09-09T01:53:44Z")

</div>

I would imagine the discussion here: [https://github.com/JuliaData/Nulls.jl/issues/6](https://github.com/JuliaData/Nulls.jl/issues/6)

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [September 9, 2017, 3:51pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/33 "2017-09-09T15:51:29Z")

</div>

> [@gabrielgellner](#):
>
> I would imagine the discussion here: [Union{T,Null} inference in structs · Issue #6 · JuliaData/Missings.jl · GitHub](https://github.com/JuliaData/Nulls.jl/issues/6)

That is the one.

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [September 9, 2017, 4:55pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/34 "2017-09-09T16:55:09Z")

</div>

I think I more or less understand why this won’t work for Query.jl design as to produce columns efficiently out of a named tuples iterator you’d need to infer the return type, which is harder with `Union{T, Null}`. However I’m still a bit confused as to what is the policy when collecting into a data structure. The following puzzles me:

```julia
julia> using DataFrames, IterableTables, IndexedTables

julia> df = DataFrame(x = [1, 2], y = rand(2))
2×2 DataFrames.DataFrame
│ Row │ x │ y │
├─────┼───┼───────────┤
│ 1 │ 1 │ 0.0905943 │
│ 2 │ 2 │ 0.220108 │

julia> t = IndexedTable(df)
x │ y
──┼──────────
1 │ 0.0905943
2 │ 0.220108

julia> typeof(t[1])
NamedTuples._NT_y{DataValues.DataValue{Float64}}

```

Did we end up with DataValues because those are the standard way of representing missing data with IndexedTables, or is it an artefact of having converted via IterableTables?

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [September 9, 2017, 8:59pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/35 "2017-09-09T20:59:46Z")

</div>

Artifact of converting via IterableTables

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [September 10, 2017, 4:32am UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/36 "2017-09-10T04:32:30Z")

</div>

> [@quinnj](#):
>
> Artifact of converting via IterableTables

Yep, that is right. [IndexedTables.jl](https://github.com/JuliaComputing/IndexedTables.jl) doesn’t have a default way to represent missing values as far as I can tell, so for now I’m just creating `Vector{DataValue{T}}` columns when you convert a source that has columns that can hold missing data. Of course for source columns that are just `Array`s you will just get an `Array` in the sink. For all the other sinks I respect whatever that sink uses for missing data, i.e. for `DataFrame` you’ll get `DataVector`s, for `DataTable` you’ll get `NullableVector`s etc. At the end of the day this is a decision each sink needs to make, i.e. how that sink wants to represent missing data. The iterable tables interface in itself doesn’t mandate/prescribe anything on that front, essentially it just interops with whatever choice the sink makes.

If the maintainers of [IndexedTables.jl](https://github.com/JuliaComputing/IndexedTables.jl) at some point pick a standard way to represent missing data in that package, it should be super easy to adjust things so that when you convert from an iterable table with missing data, you’ll get that standard data structure.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [September 12, 2017, 2:55pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/37 "2017-09-12T14:55:36Z")

</div>

I’m sure this is already in-process, but it would be really nice if all of the new versions of DataFrames, DataStreams, CSV, Feather, CategoricalArrays and whatever-else get tagged. Right now they are blocking each other and the package manager is flipping it’s shit.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [September 12, 2017, 7:23pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/38 "2017-09-12T19:23:45Z")

</div>

That’s on purpose. The new DataStreams/CSV/Feather and CategoricalArrays are incompatible with the current DataFrames, so upper bounds have been added. You need to choose either to stay on the released version of DataFrames, and the package manager will automatically use older releases of the dependencies; or to use DataFrames from git master, and update all dependencies to their most recent releases.

This situation will of course change as soon as we tag a new DataFrames release. But then it will be incompatible with all packages depending on the current DataFrames, so a similar problem will appear until they are all updated.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [September 29, 2017, 12:45pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/39 "2017-09-29T12:45:01Z")

</div>

> [@quinnj](#):
>
> [the plan, i.e. not bitarrays] supports Arrays of arbitrary dimensions, Unions with more than 2 unioned types

You can still support more than two types with a bitarray, and even with a more compressed form:

I was thinking, is a SparseArray worth it, say if most are values are not Null? It seems it could be slow, but a hybrid approach could work.

A bitarray, to check if say if you have your likely type (not Nulls or other types) for say every 16 values (just an idea, for something like a cache-line worth) in your main array.

---

<div class="post-metadata">

**Author:** ![Nosferican](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nosferican/32/9275_2.png) [@Nosferican](https://discourse.julialang.org/u/Nosferican)\
**Post date:** [November 8, 2017, 5:11pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/40 "2017-11-08T17:11:04Z")

</div>

Is there an update of when will DataFrames v0.11 will be released? I suspect many packages would like to start migrating to Julia 0.7, but have DataFrames/StatsModels dependencies. It might take a bit of work in StatsModels, but were waiting for a new DataFrames version to get tagged.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [November 8, 2017, 5:53pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/41 "2017-11-08T17:53:55Z")

</div>

You can have a look at the issues tagged for the 0.11 milestone. There’s still at least one issue to fix with CategoricalArrays, but we’re not too far. StatsModels is ready, there’s an open PR which you can use if you want to start working on a port.

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [November 8, 2017, 6:06pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646/42 "2017-11-08T18:06:11Z")

</div>

I’ve been using the master of `DataFrames` (Nulls version) for a while now, and I can say that I am quite happy. The biggest obstacle right now is that pulling the masters of everything in the ecosystem makes the package manager hysterical (hopefully the new package manager coming with 0.7 will make these sorts of things smoother in the future).

[Previous page](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646.md?page=1)

[Next page](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646.md?page=3)