# Does the concept of type-stability apply to DataFrames or Tables?

**URL:** <https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434>\
**Category:** Data\
**Tags:** dataframes, tables, type-stability\
**Created:** [October 14, 2017, 12:12pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434 "2017-10-14T12:12:58Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 14, 2017, 12:12pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/1 "2017-10-14T12:12:58Z")

</div>

I understand that if you write type stable code in Julia then it’s easier to optimize and hence will result in fast running code.

But I am thinking about this in the DataFrames context, say I have a function that takes as argument a dataframe and output a dataframe always. Now the output is type-stable since it’s alway outputing a dataframe. But I can output datarames with different columns each time. That should be type-unstable right?

I am a little confused here, does it mean that by design Julia can’t optimize code that outputs Dataframes because to do that it need to know the type of each column, but the type information is not encoded into the definition of a dataframe, hence even if a function is type-stable in that it produces a dataframe always, it will still be slow as the columns’s type may change from run to run depending on inputs?

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [October 14, 2017, 12:56pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/2 "2017-10-14T12:56:30Z")

</div>

Your question is relevant - it’s been one of the main issues with designing DataFrames in Julia. The short answer is that with the new versions of DataFrames coming out soon, they will be type-stable.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [October 14, 2017, 12:57pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/3 "2017-10-14T12:57:52Z")

</div>

Yes, that’s indeed a significant issue. The TypedTables package provides an alternative to `DataFrame` which includes column types in the table type. That ensures that specialized code can be compiled for each set of column types. This is an advantage when you work with tables which all have the same shape.

But it can also be a problem if you work with ots of very different tables, as is often the case if you select subsets of variables during various operations. In that case, you’ll have to recompile all functions each time the columns change, which could slow down operations rather than make them faster (introducing a fixed cost).

An intermediary approach, used for example by DataFrames master in the joining and grouping code, is to separate kernel functions which perform the intensive operations, and have them take a tuple of columns rather than a `DataFrame`. That way, these functions are specialized, but the code wrapping them, which wouldn’t benefit so much from specialization, isn’t. We could probably apply this approach in other places. The question is always whether the time required for compilation is worth it. The tradeoff is clearly positive when an operation can be applied separately to each column, since the number of different column types is limited. It’s less obvious when specializing on a combination of column types, since their number can be very high (and is dependent on order of columns). Only experimentation can tell.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [October 14, 2017, 12:59pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/4 "2017-10-14T12:59:19Z")

</div>

> [@mkborregaard](#):
>
> The short answer is that with the new versions of DataFrames coming out soon, they will be type-stable.

Actually, they won’t. DataFrames master isn’t very different from the current release in that regard. Use TypedTables if you need a fully type-stable approach.

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [October 14, 2017, 1:13pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/5 "2017-10-14T13:13:27Z")

</div>

😮 Oh. I can tell I’ve lost contact with what goes on in the TableVerse.

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [October 14, 2017, 1:31pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/6 "2017-10-14T13:31:30Z")

</div>

It’s not easy to keep track… I believe [IndexedTables](https://github.com/JuliaComputing/IndexedTables.jl) is also a type-stable alternative. It is true that there is a bit of recompilation cost, but if you’re interested in working with a dataset with a lot of rows, it’s probably worth it.

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [October 14, 2017, 2:14pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/7 "2017-10-14T14:14:33Z")

</div>

Isn’t the plan to use named-tuples to achieve type-stability in 0.7?

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [October 14, 2017, 2:29pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/8 "2017-10-14T14:29:58Z")

</div>

There’s no clear plan in that regard in the short term AFAIK. We have enough work on our plate (too much, actually, given the size of the team) with the port to Nulls. Many people are interested by the perspectives opened by `NamedTuple` (which is already used by Query to represent rows), though, so a few experimentations will likely happen, e.g. via alternative data frame types.

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [October 14, 2017, 2:33pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/9 "2017-10-14T14:33:57Z")

</div>

The people working on the different table implementations in Julia are truly moving mountains at the moment. This is really one of those areas in Julia where the user/developer ratio suffers, mainly because almost everybody uses tables. Thank God the group of developers are first-class.

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [October 14, 2017, 4:10pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/10 "2017-10-14T16:10:52Z")

</div>

I fully agree with @mkborregaard : the data developers are doing an insane amount of work!

Concerning “typed-ness”, I believe that one issue with the current DataFrame implementation (unless of course I’m missing something) is that it’s difficult to do `map` and `filter` on a DataFrame performantly, even though I believe those are basic manipulations. By `map` I mean something that takes a DataFrame and a function from named tuples to named tuples and outputs a DataFrame, and `filter` would take a a DataFrame and a predicate on named tuples. One can of course resort to external packages (such as Query), but even there the `@select` statement, which would be what here I call `map` has some limitations as it relies on type inference on NamedTuples to work.

Still, there has been a lot of discussion and interesting ideas on these topics and I’m curious to see what the outcome will be.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 14, 2017, 10:53pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/11 "2017-10-14T22:53:52Z")

</div>

> [@nalimilan](#):
>
> We have enough work on our plate (too much, actually, given the size of the team)

You can always hire me. 🕶

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 15, 2017, 2:27am UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/12 "2017-10-15T02:27:22Z")

</div>

From a person coming from R and SAS I think solving the uncertainty around dataframe type-unstability and doing it using an unified solution is key. Currently it’s hard for me to use Julia just for data manipulation like I can with R; partly because there are so many different dataframes packages out there.

One good thing is that there is alot of experimentation going on in Julia but for wide adoption of Julia in the data community there needs to be a stable and unified package for data maniipulation.

I think the ideas in IndexedTable, DataFrames etc should not be mutually exclusive, e.g. I do want to be able to index any column in a dataframe.

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [October 15, 2017, 5:01am UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/13 "2017-10-15T05:01:56Z")

</div>

I’m sure you will be a welcome addition to the team. You should be aware that noone working on DataFrames receive any salary to do so – it’s purely free voluntary work given as a gift to the julia community.

Everyone is well aware of the current incomplete state of table support in comparison to R, and the importance of it for the language. It’s just in development. The final result might just be even better than what R (python, Matlab, SAS) offers, IMHO.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 15, 2017, 7:30am UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/14 "2017-10-15T07:30:45Z")

</div>

Even though I would love to be full time employed and work on dataframes and related tech, I understand that my level of Julia is not “employable” yet. Anyways, in my most recent benchmark, R data.table was 3 times faster than Julia’s IndexedTable for a task. And it’s not clear how to accomplish certain tasks in Julia vs data.table, hence clearly there is a lot to do still in Julia. I have to say, it’s clear how Julia can make compute-tasks faster, but it’s not so clear how it can make dataframes tasks faster.

It would be ideal to have more resources put into Julia in terms of dataframe development, and it’s unrealistic to rely on unpaid volunteers to do long term development. Most open source development are done by paid developers I would say. Anyway, hopefully one day I will be good enough and a position will open up so I can work on it full time and quit the consulting work which is still using SAS — pain! I am working 40 hours on a consulting engagement and 20 hours+ on product development. That’s about 60+hours a week. I am trying my best to contribute to the Julia ecosystem as I already use Julia to do a number of things, but the best I can do now is to file bug reports and do the occassionaly simple PRs

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [October 15, 2017, 9:28am UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/15 "2017-10-15T09:28:12Z")

</div>

> [@xiaodai](#):
>
> It would be ideal to have more resources put into Julia in terms of dataframe development, and it’s unrealistic to rely on unpaid volunteers to do long term development. Most open source development are done by paid developers I would say.

Then you might be surprised by [this tweet](https://twitter.com/wesmckinn/status/909860120074555392) by Pandas’ creator (see also [this one](https://twitter.com/wesmckinn/status/908371915462643712)).

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 15, 2017, 10:20am UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/16 "2017-10-15T10:20:51Z")

</div>

“If you aren’t happy with the documentation, please submit some patches. We aren’t getting paid to write documentation for you”

can’t blame either side. Most panda dev are not being paid there are others benefitting from open source but not contributing back.

solution? every party that benefit from open source directly should give back in some ways. if not monetary then some well researched and helpful bug report.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [October 15, 2017, 10:24am UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/17 "2017-10-15T10:24:02Z")

</div>

> <https://twitter.com/wesmckinn/status/909854351694458880>

sad…but true

---

<div class="post-metadata">

**Author:** ![Daneel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/daneel/32/175_2.png) [@Daneel](https://discourse.julialang.org/u/Daneel)\
**Post date:** [October 16, 2017, 2:01pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/18 "2017-10-16T14:01:31Z")

</div>

One option to encourage development is one of the funding models Open Whisper Systems uses to develop their [Signal](https://www.signal.org) app, [BitHub](https://signal.org/blog/bithub/). BitHub is a combination of Bitcoin and GitHub to fund commits. Donations go to a common pot and each commit is apportioned a certain amount of Bitcoin.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [October 16, 2017, 4:03pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/19 "2017-10-16T16:03:29Z")

</div>

I think you’d be interested in reading @ChrisRackauckas’s recent [blog post](http://www.stochasticlifestyle.com/like-julia-scales-productive-insights-julia-developer/). In particular, I think Chris hits the nail on the head by saying something along the lines of “don’t expect Julia to be faster than traditional C libraries used in other languages; while Julia certainly _can_ achieve performance parity, it’s also much younger and offers many other benefits like more productive development times and abilities to scale long-term”.

In particular, `data.table` is an extremely well-established C library that has been optimized for specific operations for years. It’s a wonderful resource! It was my favorite R package (back when I used R). DataFrames is certainly not ready to compete w/ the hand-optimized code of `data.table` in it’s current form. But there are a lot of positive signs in the Julia data-processing ecosystem: JuliaDB/IndexedTables provide a DataFrame-like table that can automatically scale across julia processes, DataStreams provides completely decoupled IO operations between various data formats (i.e. you can move data between CSV, feather files, databases, DataFrames, etc. extremely fast and efficiently w/o having to load data into memory as an intermediate step), and Query.jl provides a custom “data DSL” for doing SQL/dplyr-like operations directly on a variety of data “backends”/formats.

Just those three examples are exciting and provide an example of the power and productivity of Julia and how it can influence data operations as technical computing continues to evolve. The future is exciting and I think optimizations will continue to roll in, but I’m more excited about the kind of scale and modularity that’s already being put in place in Julia (which IMO, you can’t find in other languages).

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 17, 2017, 6:54pm UTC](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434/20 "2017-10-17T18:54:14Z")

</div>

Since the subject has come up, I am curious. Is there a long term strategy to build funding for full time julia developers? Do we just hope that big finance companies become reliant on Julia and Julia Computing will hire full time devs on important packages with new money from consulting fees?

Would companies who use julia ever contribute to a larger pot for julia development like you see with the Linux kernal? Or is Julia too small of a market to get that kind of support? I mean, if the pandas developers never got to work full time, is there any hope for Julia developers doing the same?

[Next page](https://discourse.julialang.org/t/does-the-concept-of-type-stability-apply-to-dataframes-or-tables/6434.md?page=2)