# Stably indexed table

**URL:** <https://discourse.julialang.org/t/stably-indexed-table/2989>\
**Category:** General Usage\
**Created:** [March 31, 2017, 4:14pm UTC](https://discourse.julialang.org/t/stably-indexed-table/2989 "2017-03-31T16:14:32Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [March 31, 2017, 4:14pm UTC](https://discourse.julialang.org/t/stably-indexed-table/2989/1 "2017-03-31T16:14:32Z")

</div>

Out of all the modules for dataframe-like structures (is there a list?), is there one where `table[:col]` or `table[Val{:col}]` is type-stable? As JMW put it:

> A second possible solution is to generate custom DataFrame types for every distinct DataFrame object. This could convert DataFrames from black-box containers that contain objects of arbitrary type into fully typed containers that can only contain objects of types that are fully known to the compiler.

Was that ever attempted?

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [March 31, 2017, 5:01pm UTC](https://discourse.julialang.org/t/stably-indexed-table/2989/2 "2017-03-31T17:01:28Z")

</div>

Take a look at [https://github.com/FugroRoames/TypedTables.jl](https://github.com/FugroRoames/TypedTables.jl).

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [March 31, 2017, 10:09pm UTC](https://discourse.julialang.org/t/stably-indexed-table/2989/3 "2017-03-31T22:09:53Z")

</div>

Awesome, thank you. @andyferris Is this package still maintained, or at least, will PRs be considered?

---

<div class="post-metadata">

**Author:** ![tshort](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tshort/32/43_2.png) [@tshort](https://discourse.julialang.org/u/tshort)\
**Post date:** [April 1, 2017, 12:35am UTC](https://discourse.julialang.org/t/stably-indexed-table/2989/4 "2017-04-01T00:35:00Z")

</div>

There’s also IndexedTables (still in development):

[https://github.com/JuliaComputing/IndexedTables.jl](https://github.com/JuliaComputing/IndexedTables.jl)

And the CompositeDataFrame (never used much) in DataFramesMeta:

[https://github.com/JuliaStats/DataFramesMeta.jl](https://github.com/JuliaStats/DataFramesMeta.jl)

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [April 1, 2017, 12:58am UTC](https://discourse.julialang.org/t/stably-indexed-table/2989/5 "2017-04-01T00:58:42Z")

</div>

And if you want to use all of these at once and be able to easily convert from one to the other etc., check out [https://github.com/davidanthoff/IterableTables.jl](https://github.com/davidanthoff/IterableTables.jl). For most of the support table types, you can then just do `IndexedTable(df)` where e.g. might be a `DataFrame` (or any of the other types supported) and things will automatically convert. Essentially this should work in all permutations of source and sink type.

For now you’ll need to use `master` of SimpleTraits, once there is a new tagged version of that I’ll register and release `IterableTables` properly.

---

<div class="post-metadata">

**Author:** ![tshort](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tshort/32/43_2.png) [@tshort](https://discourse.julialang.org/u/tshort)\
**Post date:** [April 1, 2017, 2:15am UTC](https://discourse.julialang.org/t/stably-indexed-table/2989/6 "2017-04-01T02:15:39Z")

</div>

Cool! Hadn’t seen that.

---

<div class="post-metadata">

**Author:** ![andyferris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andyferris/32/235_2.png) [@andyferris](https://discourse.julialang.org/u/andyferris)\
**Post date:** [April 1, 2017, 2:24am UTC](https://discourse.julialang.org/t/stably-indexed-table/2989/7 "2017-04-01T02:24:32Z")

</div>

PRs - always, definitely 😉

I have been musing about slightly more flexible data containers (that support things like group by operations, column and row indices that can be static or dynamic, and fast lookup indexes). But so far I don’t have anything meaningful and open source to release (for work, I do have a bespoke strongly typed tabular data structure with fast spatial lookup, which has a more convenient interface than in TypedTables.)

Anyway, I still believe that, in the right circumstances, a TypedTable is really useful and convenient, so  
future work on TypedTables to make it have a better API, be more flexible and fully featured and work with other packages, and to fix bugs seems worthwhile.

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [April 1, 2017, 2:59am UTC](https://discourse.julialang.org/t/stably-indexed-table/2989/8 "2017-04-01T02:59:20Z")

</div>

IndexedTables is interesting, but unless I’m mistaken, it’s not really a tabular type. It looks more like a dictionary / map.

I just tried DataTables.jl and Query.jl for the first time:

```julia
dt = DataTable(A = 1:4, B = ["M", "F", "F", "M"])
f(dt) = @from i in dt begin
           @select i.A
           @collect
       end
@code_warntype f(dt)

```

tells me that this is not type-stable. Am I correct that the point of Query.jl and DataFramesMeta.jl’s `@with` is that whatever computation happens inside of their respective macro expansions will be done in a type-stable and efficient way, but I shouldn’t expect a type-inferred return value?

Regarding `CompositeDataFrame`, is there a substantive difference with `TypedTables`? My understanding is that the former generates a type by calling `eval`, while the latter uses parametric types to achieve essentially the same thing (which is perhaps a bit cleaner?) I would expect both approaches to have essentially the same performance characteristics, right?

> [@davidanthoff](#):
>
> And if you want to use all of these at once and be able to easily convert from one to the other etc., check out [GitHub - queryverse/IterableTables.jl: Implementations of the TableTraits.jl interface for various packages · GitHub](https://github.com/davidanthoff/IterableTables.jl)

Awesome! Is there still any hope of a common interface package, so that we don’t have to use `Requires.j`? AbstractTables.jl looks pretty dead.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [April 1, 2017, 4:27am UTC](https://discourse.julialang.org/t/stably-indexed-table/2989/9 "2017-04-01T04:27:05Z")

</div>

> Am I correct that the point of Query.jl and DataFramesMeta.jl’s @with is that whatever computation happens inside of their respective macro expansions will be done in a type-stable and efficient way, but I shouldn’t expect a type-inferred return value?

For Query, yes. Not sure about `DataFramesMeta`. I am thinking about something like this for Query currently:

```julia
@from i in dt begin
    @select i.A
    @collect
end q begin
    # In this block q holds the result of the query, and everything is type stable
    println(q)
end

```

But I’m not sure how much I like that, so any feedback would be welcome.

> Awesome! Is there still any hope of a common interface package, so that we don’t have to use Requires.j? AbstractTables.jl looks pretty dead.

The use of `Requires.jl` is really just temporary. My goal is to move out all the source and sink specific code into the actual source and sink packages, so that `IterableTable` essentially only has [this](https://github.com/davidanthoff/IterableTables.jl/blob/master/src/IterableTables.jl) and maybe [this](https://github.com/davidanthoff/IterableTables.jl/blob/master/src/utilities.jl) code. But for now it makes things easier for me to have all the code in one repo while I iterate on things. At the end of the day `IterableTable` is an alternative to something like `AbstractTables`. The main difference really is that a) it only provides a row iteration interface (no column based interface) and b) it is not based on some common supertype, but uses traits. The latter makes the whole system much, much more flexible.

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [April 1, 2017, 2:22pm UTC](https://discourse.julialang.org/t/stably-indexed-table/2989/10 "2017-04-01T14:22:10Z")

</div>

> [@davidanthoff](#):
>
> But I’m not sure how much I like that, so any feedback would be welcome.

I think it looks OK. This also parses:

```julia
@from(i in dt, begin
    @select i.A
    @collect
end) do q
    # In this block q holds the result of the query, and everything is type stable
    println(q)
end

```

> [@davidanthoff](#):
>
> The use of Requires.jl is really just temporary. My goal is to move out all the source and sink specific code into the actual source and sink packages, so that IterableTable essentially only has this and maybe this code.

Looking forward to that, good luck!

---

<div class="post-metadata">

**Author:** ![tshort](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tshort/32/43_2.png) [@tshort](https://discourse.julialang.org/u/tshort)\
**Post date:** [April 1, 2017, 6:52pm UTC](https://discourse.julialang.org/t/stably-indexed-table/2989/11 "2017-04-01T18:52:50Z")

</div>

IndexedTables has a Columns type you can use as a table. Its API is still a little clunkier than a DataFrame, but it’s type stable.