# \[ANN\] RowTables.jl

**URL:** <https://discourse.julialang.org/t/ann-rowtables-jl/12608>\
**Category:** Data\
**Tags:** announcement\
**Created:** [July 23, 2018, 3:08pm UTC](https://discourse.julialang.org/t/ann-rowtables-jl/12608 "2018-07-23T15:08:58Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [July 23, 2018, 3:08pm UTC](https://discourse.julialang.org/t/ann-rowtables-jl/12608/1 "2018-07-23T15:08:58Z")

</div>

`RowTables.jl` is intended to be (and often is) faster than `DataFrames` at row-wise operations.

[https://github.com/jlapeyre/RowTables.jl](https://github.com/jlapeyre/RowTables.jl)

This package is not registered. See `README.md` for a few benchmarks.

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [July 25, 2018, 2:35am UTC](https://discourse.julialang.org/t/ann-rowtables-jl/12608/2 "2018-07-25T02:35:18Z")

</div>

Very cool! Alas, I almost never have data where I do only row-wise stuff or only column-wise stuff. It’s always a mix…

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [July 25, 2018, 8:02am UTC](https://discourse.julialang.org/t/ann-rowtables-jl/12608/3 "2018-07-25T08:02:21Z")

</div>

IndexedTables should be quite fast at iterating rows, esp. if the entries are `isbits`. Have you benchmarked your package relative to it?

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [July 25, 2018, 12:45pm UTC](https://discourse.julialang.org/t/ann-rowtables-jl/12608/4 "2018-07-25T12:45:47Z")

</div>

The idea is that if you have some row-intensive operations, eg, pushing rows, or permuting (repeatedly), it is worth it to convert to `RowTable`, do the operations and convert back. The conversion is just `RowTable(df, tuples=false)` and `DataFrame(rt)`. I made the indexing almost the same as `DataFrames` (the exception is in the examples in the Readme). It’s not a problem to make it exactly the same. For this reason, it might be easy to try the optimization by adding a couple of lines and to revert.

I also worked on it because I was curious.

I wrote this several months ago. Benchmarking showed that it was not worth the conversion in the use case at hand, so I left it. A couple of days ago, I thought, why not post the code, so I wrote a README, etc. and put it on github.

I did new benchmarks for the README. I don’t have a use for it now, but the benchmark times make me think such a case might arise.

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [July 25, 2018, 1:08pm UTC](https://discourse.julialang.org/t/ann-rowtables-jl/12608/5 "2018-07-25T13:08:33Z")

</div>

I just tried this a little bit. Doing it more carefully would take a bit of time.  
I had trouble testing `IndexedTables` when I wrote this several months ago, don’t remember what it was. I just now installed `IndexedTables`. AFAICT, it requires v0.6. But, `RowTables` only works with v0.7. Julia version is a major factor that I can’t hold constant.

I just spent some time trying to understand `IndexedTables`. I only looked at the non-sparse storage option. I have not worked much with named Tuples. Trying construct them programatically was a PITA… I’m sure there are better ways. To try to make the comparison more fair, I store the `RowTable` rows as `Tuples`. Making them named tuples would take a bit of time for me, probably very little for someone who knows them well.

In any case, once you have the row, quick benchmarks show that, say summing the elements, takes the same time with plain-vanilla Tuples on v0.7 as for named Tuples on v0.6 **[No, see below]**. It looks like `IndexedTables` stores the data as columns. It requires a fixed named Tuple type. To get a row, it creates a named tuple by iterating over columns. As expected, accessing a row is faster with `RowTables`, with the advantage increasing with the number of columns. But, in some code, I found that `IndexedTables` got rows faster; I couldn’t chase this down quickly. In any case, if you have a 50x100 table, and sum all rows by summing each row and accumulating, then `RowTables` is faster. I won’t quote numbers now because there are too many variables to nail down.

If someone is interested, I’d be more motivated (Well, I’m motivated, I just have other stuff to do.) to do benchmarks and post them.

EDIT: Named tuples are not only named, but typed, and the compiler takes advantage of this. Summing a named tuple of floats is faster than summing an ordinary tuple with the same data.

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [July 25, 2018, 7:02pm UTC](https://discourse.julialang.org/t/ann-rowtables-jl/12608/6 "2018-07-25T19:02:38Z")

</div>

> [@piever](#):
>
> IndexedTables should be quite fast at iterating rows, esp. if the entries are `isbits` . Have you benchmarked your package relative to it?

The README shows benchmarks compared to `DataFrames`. I benchmarked `IteratedTables` and the numbers are similar. It’s not surprising. One package stores data by rows, the other two by columns. I added support for named tuples to `RowTables`. But, this is not worth much without associated functionality… like type-stable column access. There are several table packages, and they require a lot of effort to maintain. Likewise, developing `RowTables` would require a lot of effort. `RowTables` is an experiment to see how performant row-wise storage is.

---

<div class="post-metadata">

**Author:** ![zhangliye](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zhangliye/32/3208_2.png) [@zhangliye](https://discourse.julialang.org/u/zhangliye)\
**Post date:** [July 26, 2018, 2:40am UTC](https://discourse.julialang.org/t/ann-rowtables-jl/12608/7 "2018-07-26T02:40:48Z")

</div>

The benchmark looks very impressive! Most of my operation in the work is to read the data and do some calculation. Very useful package!