# Query.jl: How can I write type-preserving methods for iterable tables

**URL:** <https://discourse.julialang.org/t/query-jl-how-can-i-write-type-preserving-methods-for-iterable-tables/19956>\
**Category:** Data\
**Tags:** question\
**Created:** [January 23, 2019, 1:29am UTC](https://discourse.julialang.org/t/query-jl-how-can-i-write-type-preserving-methods-for-iterable-tables/19956 "2019-01-23T01:29:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [January 23, 2019, 1:29am UTC](https://discourse.julialang.org/t/query-jl-how-can-i-write-type-preserving-methods-for-iterable-tables/19956/1 "2019-01-23T01:29:27Z")

</div>

In MLJ.jl, we are trying to write a data-agnostic machine-learning framework and are presently trying to make this work using Query, to which I am new.

We would like to write methods that take an iterable table as input and that output a table _of identical type_ (assuming the type is sink-supported). For example, a function to standardise the numerical features (columns) of some table, or a function to project the table onto a smaller training set of rows. If I give my method a `DataFrame`, I want a `DataFrame` as output. If I give it a `TypedTable`, then the output should be a `TypedTable`.

Here’s my attempt at a function to select a subset of columns from some Query utterable table, returning an object of the same type:

```julia
function getcols(X::T, c::AbstractArray{I}) where {T,I<:Union{Symbol,Integer}}

    TableTraits.isiterabletable(X) || error("Argument is not an iterable table.")

    row_iterator = @from row in X begin
        @select project(row, c)
        @collect T
    end
                    
end

```

Here `project(row, c)` is just the projection of the named tuple `row` onto a named tuple with only those labels/indices specified by `c`.

Now `getcols` works as for a `DataFrame` but not for, say a `TypedTable`. The problem is that `TypedTable` has type parameters which will be different for the output than the input. In other words, `T` in this case is `TypedTables.Table{NamedTuple{(:x1, :x2........` for the signature, but `T` in the collect statement needs to be just `TypedTable` to work (I guess).

So what’s a way to do this that works?

---

<div class="post-metadata">

**Author:** ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)\
**Post date:** [January 28, 2019, 12:58am UTC](https://discourse.julialang.org/t/query-jl-how-can-i-write-type-preserving-methods-for-iterable-tables/19956/2 "2019-01-28T00:58:43Z")

</div>

BTW: In Tables.jl there is a way to do this using the materialiser method.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [February 13, 2019, 5:46am UTC](https://discourse.julialang.org/t/query-jl-how-can-i-write-type-preserving-methods-for-iterable-tables/19956/3 "2019-02-13T05:46:16Z")

</div>

Sorry for not catching this earlier, somehow this slipped by.

In general the various [Query.jl](https://github.com/queryverse/Query.jl) operators quite consciously avoid this kind of design, where the output type in some way is determined by the input type. The “why” is probably easier to see with the method piping syntax:

```julia
source |> @select(:colA, :colB) |> @filter(_.colA=="foo") |> DataFrame

```

This is a pretty canonical query, expressed as a pipeline. Now, one could of course design this such that `@select` returns a `DataFrame` if `source` is a `DataFrame` etc. But that would really be quite inefficient: here we want to make sure that the projection and the filter get fused into one loop only, and that we don’t allocate an intermediate `DataFrame` that then gets passed onto the `@filter` operation.

So, instead, _all_ operations in [Query.jl](https://github.com/queryverse/Query.jl) are lazy. If you just run `source |> @select(:colA, :colB) |> @filter(_.colA=="foo")`, _no_ data is actually touched, all we are constructing is a chained series of lazy iterators that _can_ be executed at some point, and then everything gets fused. What triggers execution of this query is that it gets passed to `DataFrame` (or some other table type, or a simple `collect` call).