# Table views

**URL:** <https://discourse.julialang.org/t/table-views/45834>\
**Category:** Data\
**Tags:** api, tables\
**Created:** [August 31, 2020, 1:56pm UTC](https://discourse.julialang.org/t/table-views/45834 "2020-08-31T13:56:35Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [August 31, 2020, 1:56pm UTC](https://discourse.julialang.org/t/table-views/45834/1 "2020-08-31T13:56:36Z")

</div>

There are various efforts in the tables ecosystem to create a common language for table operations:

- [Query.jl](https://github.com/queryverse/Query.jl)
- [DataKnots.jl](https://github.com/rbt-lang/DataKnots.jl)
- [TableOperations.jl](https://github.com/JuliaData/TableOperations.jl)
- [Volcanito.jl](https://github.com/johnmyleswhite/Volcanito.jl)

I couldn’t find in any of these efforts an operation to select rows of a table based on indices in a lazy fashion. Is there something conceptual that I am missing about taking views of tables in the vertical dimension? Why none of these efforts provides a `view(table, rows, cols)` as a generalization of `select(table, cols)`?

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [August 31, 2020, 2:38pm UTC](https://discourse.julialang.org/t/table-views/45834/2 "2020-08-31T14:38:48Z")

</div>

I mean DataFrames.jl already works. If anything it would be in Tables.jl

```julia
using DataFrames

df = DataFrame(A = 1:3, B = [2, 1, 2])

@view df[1,2] #this already works

```

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [August 31, 2020, 2:55pm UTC](https://discourse.julialang.org/t/table-views/45834/3 "2020-08-31T14:55:09Z")

</div>

If you are returning columns with `Tables.getcolumns` you don’t have to return a materialized column, just any object that can be indexed. Is that what you mean?

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [August 31, 2020, 3:04pm UTC](https://discourse.julialang.org/t/table-views/45834/4 "2020-08-31T15:04:07Z")

</div>

At least in the case of Tables.jl/TableOperations.jl (and Query.jl actually), their interfaces are “more generic” than allowing row sub-setting. All you have to provide are row iterators, which means the table _doesn’t_ (and can’t in some cases, e.g. w/ forward-only row streaming tables) have to implement arbitrary random-access indexing/view capabilities. I’ve said elsewhere that I think it’d be useful to have an IndexableTables.jl interface that added this requirement (i.e. tables must support random-access) and then you could define a generic row-indexing/view operation on any IndexableTable. Just takes someone to sit down and write out all the details, and help do some implementations to get something working.

Personally, I find I can do the work I’m interested in _without_ needing views over rows, so I haven’t been highly motivated to work on such an interface. I find I can always change my data processing operations to either work on a stream of rows (i.e. `Tables.rows`), or materialized as entire columns which _do_ support indexing in the Tables.jl interface (i.e. `Tables.columns` returns a set of indexable columns).

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [August 31, 2020, 3:15pm UTC](https://discourse.julialang.org/t/table-views/45834/5 "2020-08-31T15:15:36Z")

</div>

Thank you all. I think the issue was pointed out perfectly by @quinnj: we lack a _indexable_ table interface for which views over rows are permitted.

In my applications the tables are always finite and we know a priori the number of rows. They are not necessarily loaded into memory, but we would like to query the i-th element in the database as needed for some local computation.

For now, I think I will have to materialize the table as columns `Tables.columns` and then slice the columns vertically.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [August 31, 2020, 3:24pm UTC](https://discourse.julialang.org/t/table-views/45834/6 "2020-08-31T15:24:42Z")

</div>

This is what I’ve done with DimensionalData.jl for similar reasons. Tables.getcolums are lazy `AbstractVector`s so I don’t have to allocate full columns for each dimension index. The array data is just reshaped with `vec` as it’s in-memory.

What will be interesting is doing this with disk-based data in GeoData.jl. It would be good to hook into the chunking system in DiskArrays.jl to read out the table to match the disk chunks.

What are the data sources you want to load lazily? I’m assuming this is for GeoStats.jl

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [August 31, 2020, 3:32pm UTC](https://discourse.julialang.org/t/table-views/45834/7 "2020-08-31T15:32:43Z")

</div>

The requirements I have in mind have to do with the fact that many partitioning algorithms should return lazy views of the table rows as opposed to copying chunks of the table in different spatial regions. Later on, this “lazy loading from disk” issue will also be important, but I am just trying to get the internal algorithms working without copies of the properties.

I will try to wrap the internal table of the spatial data in GeoStats.jl in a struct, and then trigger the slice on `Tables.columns` when necessary.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [August 31, 2020, 5:37pm UTC](https://discourse.julialang.org/t/table-views/45834/8 "2020-08-31T17:37:24Z")

</div>

This is the function I introduced temporarily for the views:

```julia
# helper function for table view
function viewtable(table, rows, cols)
  t = Tables.columns(table)
  v = map(cols) do c
    col = Tables.getcolumn(t, c)
    c => view(col, rows)
  end
  (; v...)
end

```

It returns a “column table” according to the Tables.jl docs. It is lazy in the sense that I am taking views of columns and returning these views in a named tuple. The function starts by assuming that `Tables.columns` is available, i.e. `Tables.columnaccess(table) == true`. It can be made more robust by treating the other case with `Tables.rowaccess(table) == true` where one could collect the rows and return a view.