# ANN: IterableTables.jl - a lightweight abstract table interface

**URL:** <https://discourse.julialang.org/t/ann-iterabletables-jl-a-lightweight-abstract-table-interface/3606>\
**Category:** Community\
**Tags:** package, announcement\
**Created:** [May 8, 2017, 11:51pm UTC](https://discourse.julialang.org/t/ann-iterabletables-jl-a-lightweight-abstract-table-interface/3606 "2017-05-08T23:51:33Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [May 8, 2017, 11:51pm UTC](https://discourse.julialang.org/t/ann-iterabletables-jl-a-lightweight-abstract-table-interface/3606/1 "2017-05-08T23:51:33Z")

</div>

Hi all,

I just tagged version v0.0.2 of [IterableTables.jl](https://github.com/davidanthoff/IterableTables.jl). IterableTables is a lightweight abstract table interface that makes it easier to convert between different table types. It also enables the use of many table types in situations that traditionally required a `DataFrame`. Finally, it integrates tightly with [Query.jl](https://github.com/davidanthoff/Query.jl), so that any iterable table can be queried using that package.

# Overview

The package implements the iterable tables interface for the following types/packages: [DataFrames](https://github.com/JuliaStats/DataFrames.jl), [DataStreams](https://github.com/JuliaData/DataStreams.jl) (including [CSV](https://github.com/JuliaData/CSV.jl), [Feather](https://github.com/JuliaStats/Feather.jl), [SQLite](https://github.com/JuliaDB/SQLite.jl), [ODBC](https://github.com/JuliaDB/ODBC.jl)), [DataTables](https://github.com/JuliaData/DataTables.jl), [IndexedTables](https://github.com/JuliaComputing/IndexedTables.jl),  
[TimeSeries](https://github.com/JuliaStats/TimeSeries.jl), [TypedTables](https://github.com/FugroRoames/TypedTables.jl), [DifferentialEquations](https://github.com/JuliaDiffEq/DifferentialEquations.jl) (any `DESolution`) and any iterator who produces elements of type [NamedTuple](https://github.com/blackrock/NamedTuples.jl). Essentially all of these types are iterable tables once you load `IterableTables` and can be passed to any function or constructor that expects an iterable table.

Iterable tables can be consumed by a number of functions. In `IterableTables` the functions that can consume iterable tables can broadly be grouped into a) constructors of table types and b) functions that traditionally expected a `DataFrame ` argument.

`IterableTables` adds constructors that expect an iterable table to the following types: [DataFrames](https://github.com/JuliaStats/DataFrames.jl) [DataTables](https://github.com/JuliaData/DataTables.jl), [IndexedTables](https://github.com/JuliaComputing/IndexedTables.jl), [TimeSeries](https://github.com/JuliaStats/TimeSeries.jl) and [TypedTables](https://github.com/FugroRoames/TypedTables.jl). If you call any of these constructors and pass an iterable table as the argument, the package will create an instance of that respective type and copy the data from the argument into that new instance you just created.

`IterableTables` adds methods that expect an iterable table to a number of functions in various packages. These include: all the modeling functions in [DataFrames](https://github.com/JuliaStats/DataFrames.jl) and [StatsModels](https://github.com/JuliaStats/StatsModels.jl) (for example you can now run a linear regression on data stored in a `DataTable` using the [GLM](https://github.com/JuliaStats/GLM.jl) package). It also adds methods to [Gadfly](https://github.com/GiovineItalia/Gadfly.jl) and [VegaLite](https://github.com/fredo-dedup/VegaLite.jl) that allow you to plot any iterable table, not just `DataFrame`s. Finally it integrates with the DataStreams stack so that you can write out any iterable table as either a [CSV](https://github.com/JuliaData/CSV.jl) or [Feather](https://github.com/JuliaStats/Feather.jl) file.

The integration with [Query](https://github.com/davidanthoff/Query.jl) allows you to query an iterable table source, and you can materialize any query that creates a table like result into any of the types that have a constructor that accepts an iterable table.

# Examples

Lets say you start with a `DataFrame`:

```julia
using DataFrames

df = DataFrame(Name=["John", "Sally", "Jim"], Age=[34.,25.,67.], Children=[2,0,3])

```

Using `IterableTables` you can easily convert this `DataFrame` into many other table types:

```julia
using DataTables, TypedTables, IterableTables

# Convert to a DataTable
dt = DataTable(df)

# Convert to a TypedTable
tt = Table(df)

```

These conversions work in any direction, i.e. you could also have started with a `DataTable` or any of the other supported table types.

The integration with packages like Gadfly or the statistical functionality is equally simple. For example, to run a regression on the `TypedTable` we just created, you simple would write

```julia
using GLM

# Run a regression on a TypedTable
lm(@formula(Children~Age),tt)

```

Or to plot the `DataTable` we created you would do:

```julia
using Gadfly

# Plot a DataTable
plot(dt, x=:Age, y=:Children, Geom.line)

```

Finally, this is completey integrated with Query. For example, the following query starts with a `TypedTable` and the materializes the results into a `DataTable`:

```julia
new_dt = @from i in tt begin
    @where i.Age > 30
    @select {i.Name, i.Children}
    @collect DataTable
end

```

# Development status

The package should be fairly stable at this point, it has a pretty comprehensive test suite and has a decent documentation. The various integrations have different levels of polish, i.e. some are highly optimized (`DataTables`) while others could use some performance work. For now I want to keep all the integration code inside the `IterableTables` package, mainly because there might be one big change coming in how I handle missing data and it will be a lot easier if the integration code is not spread out over many different packages. Long term I hope to convince the maintainers of the various integrated packages to move the code that is specific to their packages into their package code base.

If you have a package where you either want to consume an iterable table or you would like to expose your own type as an iterable table, please open an issue in the [IterableTables](https://github.com/davidanthoff/IterableTables.jl) repository, so that we can figure out the best way to achieve that.

And of course, as always, any help with the package would be greatly appreciated!

---

<div class="post-metadata">

**Author:** ![evanfields](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/evanfields/32/1744_2.png) [@evanfields](https://discourse.julialang.org/u/evanfields)\
**Post date:** [May 9, 2017, 12:16am UTC](https://discourse.julialang.org/t/ann-iterabletables-jl-a-lightweight-abstract-table-interface/3606/2 "2017-05-09T00:16:47Z")

</div>

Wow. Thanks for the package. I have for a while been vaguely aware that the data ecosystem is more than just DataFrames and I should check out the “anthoffverse” but reading the overview here has really moved that to the top of the priority list!