# Tables.jl: columntable to rowtable

**URL:** <https://discourse.julialang.org/t/tables-jl-columntable-to-rowtable/36112>\
**Category:** Data\
**Created:** [March 17, 2020, 8:57pm UTC](https://discourse.julialang.org/t/tables-jl-columntable-to-rowtable/36112 "2020-03-17T20:57:55Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [March 17, 2020, 8:57pm UTC](https://discourse.julialang.org/t/tables-jl-columntable-to-rowtable/36112/1 "2020-03-17T20:57:55Z")

</div>

I’m trying to implement an interface to a column-based table storage backend, but struggle to make it work efficiently. The minimal implementation looks like this:

```julia
import Tables

struct MyTable <: Tables.AbstractColumns
end

Tables.istable(::MyTable) = true
Tables.columnaccess(::MyTable) = true
Tables.columns(table::MyTable) = table

Tables.columnnames(table::MyTable) = [:A, :B, :C]
function Tables.getcolumn(table::MyTable, colname::Symbol)
    sleep(1) # simulate loading the column from somewhere
    return collect(1:10)
end

```

It works, but is much slower than should be: i.e., taking the first row takes 4 seconds whereas it could be done in 3:

```julia
@time display(MyTable() |> Tables.rows |> first)

Tables.ColumnsRow{MyTable}:
 :A 1
 :B 1
 :C 1
  4.148632 seconds (246.80 k allocations: 12.727 MiB)

```

It’s even worse if I try to convert a `MyTable` to a `rowtable` - a `Vector` of `NamedTuple`s:

```julia
@time MyTable() |> Tables.rowtable

 31.176221 seconds (338.99 k allocations: 17.856 MiB)
10-element Array{NamedTuple{(:A, :B, :C),Tuple{Int64,Int64,Int64}},1}:
 (A = 1, B = 1, C = 1)   
 (A = 2, B = 2, C = 2)   
 (A = 3, B = 3, C = 3)   
 (A = 4, B = 4, C = 4)   
 (A = 5, B = 5, C = 5)   
 (A = 6, B = 6, C = 6)   
 (A = 7, B = 7, C = 7)   
 (A = 8, B = 8, C = 8)   
 (A = 9, B = 9, C = 9)   
 (A = 10, B = 10, C = 10)

```

That is, `Tables` loads every whole column every time it needs to access a row.

The only time-efficient way to convert `MyTable` to a `rowtable` I found is going through a `DataFrame`:

```julia
@time MyTable() |> DataFrame |> Tables.rowtable

  3.052622 seconds (129.00 k allocations: 6.975 MiB)
10-element Array{NamedTuple{(:A, :B, :C),Tuple{Int64,Int64,Int64}},1}:
 (A = 1, B = 1, C = 1)   
 (A = 2, B = 2, C = 2)   
 (A = 3, B = 3, C = 3)   
 (A = 4, B = 4, C = 4)   
 (A = 5, B = 5, C = 5)   
 (A = 6, B = 6, C = 6)   
 (A = 7, B = 7, C = 7)   
 (A = 8, B = 8, C = 8)   
 (A = 9, B = 9, C = 9)   
 (A = 10, B = 10, C = 10)

```

Even faster than getting just the first row using `Tables` API!

So, my question is - am I missing something here? How it is supposed to work efficiently?

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [May 3, 2020, 2:33pm UTC](https://discourse.julialang.org/t/tables-jl-columntable-to-rowtable/36112/2 "2020-05-03T14:33:12Z")

</div>

Any ideas of what is wrong in my usage of `Tables` API? How is it supposed to work for such a usecase?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [May 3, 2020, 5:04pm UTC](https://discourse.julialang.org/t/tables-jl-columntable-to-rowtable/36112/3 "2020-05-03T17:04:58Z")

</div>

You need to implement `Tables.rows` yourself to make this work.

```julia
function Tables.rows(m::MyTable)
    Tables.rows(NamedTuple(m))
end

```

This will bring it down to 3 seconds, just the time it takes to call `getcolumn` 3 times.

I’m not sure your ultimate goals here, but I think that when you initialize your `MyTable` type I wonder if you should do all your `collect`ing there and pay a large upfront cost on creating the table rather than pay it over and over again whenever `getcolumns` gets called.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [May 3, 2020, 5:34pm UTC](https://discourse.julialang.org/t/tables-jl-columntable-to-rowtable/36112/4 "2020-05-03T17:34:54Z")

</div>

I though about this possibility of loading all the data every time, but in this case I lose the “lazy” access to columns - they will always be loaded, even if I want to access a specific column this time.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [May 3, 2020, 5:53pm UTC](https://discourse.julialang.org/t/tables-jl-columntable-to-rowtable/36112/5 "2020-05-03T17:53:39Z")

</div>

Hmmm… yeah I’m not sure about how to approach this. In theory Tables.jl should work well for lazy tables, but I don’t know how to implement it exactly. I think “lazyness” is more for iteration of rows rather than columns.

My first thought is to have an `addcolumn!` function which materializes a new column as needed, then have an `update!` method that searches and updates all the columns from your source as needed. But I don’t have a lot of experience with this.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [May 4, 2020, 4:42am UTC](https://discourse.julialang.org/t/tables-jl-columntable-to-rowtable/36112/6 "2020-05-04T04:42:28Z")

</div>

> [@aplavin](#):
>
> It works, but is much slower than should be: i.e., taking the first row takes 4 seconds whereas it could be done in 3:
> 
> ```julia-auto
> 
> ```

This is because the first column is accessed once when `Tables.rows` is called to determine the length of all columns; then each column is accessed sequentially. So not really a surprise here? If you really wanted this to just be 3 seconds, you could overload `Tables.rowcount(::MyTable)` directly to avoid [the generic definition being called](https://github.com/JuliaData/Tables.jl/blob/53dfc6c6d63f7f7aaee9ad4f4e52d447de057459/src/fallbacks.jl#L8).

> [@aplavin](#):
>
> That is, `Tables` loads every whole column every time it needs to access a row.

This is again correct, as can be seen [here](https://github.com/JuliaData/Tables.jl/blob/53dfc6c6d63f7f7aaee9ad4f4e52d447de057459/src/fallbacks.jl#L26). It’s certainly expected that `Tables.getcolumn(table, colname)` is O(1) and we should enhance the documentation as such.

> [@aplavin](#):
>
> Even faster than getting just the first row using `Tables` API!

Again, this is not that surprising, because you’re paying the 3-second cost to access each column once to put inside a DataFrame, and then a DataFrame is doing the normal O(1) iteration through rows with views that is fast.

I’m not totally clear on where the disconnect is here, but I have a guess. If I were making a Tables.jl interface implementation to a columnar-storage format, I would ensure that `Tables.getcolumn(table, colname)` returned some kind of lazy `AbstractArray`, thus ensuring `getcolumn` is very fast, and then I just have to ensure `getindex(::MyLazyColumn, i::Int)` actually materializes the data. That should allow you to use laziness and still rely on the default `Tables.rows` definitions provided by Tables.jl. Otherwise, as was suggested, you can just implement your own `Tables.rows` definition to ensure columns are only materialized once and “row views” are iterated. Happy to help chat strategies here if you can provide additional information or have other questions.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [May 4, 2020, 7:58pm UTC](https://discourse.julialang.org/t/tables-jl-columntable-to-rowtable/36112/7 "2020-05-04T19:58:01Z")

</div>

Thanks a lot for this thorough explanation! The major part I missed before was basically this:

> If I were making a Tables.jl interface implementation to a columnar-storage format, I would ensure that `Tables.getcolumn(table, colname)` returned some kind of lazy `AbstractArray` , thus ensuring `getcolumn` is very fast, and then I just have to ensure `getindex(::MyLazyColumn, i::Int)` actually materializes the data. That should allow you to use laziness and still rely on the default `Tables.rows` definitions provided by Tables.jl.