# \[ANN\] DataKnots.jl - an extensible, practical and coherent algebra of query combinators

**URL:** <https://discourse.julialang.org/t/ann-dataknots-jl-an-extensible-practical-and-coherent-algebra-of-query-combinators/22423>\
**Category:** Package Announcements\
**Tags:** announcement, data, query\
**Created:** [March 27, 2019, 5:27pm UTC](https://discourse.julialang.org/t/ann-dataknots-jl-an-extensible-practical-and-coherent-algebra-of-query-combinators/22423 "2019-03-27T17:27:58Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![xitology](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xitology/32/7570_2.png) [@xitology](https://discourse.julialang.org/u/xitology)\
**Post date:** [March 27, 2019, 5:27pm UTC](https://discourse.julialang.org/t/ann-dataknots-jl-an-extensible-practical-and-coherent-algebra-of-query-combinators/22423/1 "2019-03-27T17:27:58Z")

</div>

> **[GitHub - MechanicalRabbit/DataKnots.jl: an extensible, practical and coherent...](https://github.com/MechanicalRabbit/DataKnots.jl)**
>
> an extensible, practical and coherent algebra of query combinators - GitHub - MechanicalRabbit/DataKnots.jl: an extensible, practical and coherent algebra of query combinators

[DataKnots](https://github.com/rbt-lang/DataKnots.jl) implements an algebraic query interface of [Query Combinators](https://arxiv.org/abs/1702.08409). This algebra’s elements, or queries, represent relationships among class entities and data types. This algebra’s operations, or combinators, are applied to construct query expressions.

We seek to prove that this query algebra has significant advantages over the state of the art:

- DataKnots is a practical alternative to SQL with a declarative syntax; this makes it suitable for use by domain experts.
- DataKnots’ data model handles nested and recursive structures (unlike DataFrames or SQL); this makes it suitable for working with CSV, JSON, XML, and SQL databases.
- DataKnots has a formal semantic model based upon monadic composition; this makes it easy to reason about the structure and interpretation of queries.
- DataKnots is a combinator algebra (like XPath but unlike LINQ or SQL); this makes it easier to assemble queries dynamically.
- DataKnots is fully extensible with Julia; this makes it possible to specialize it into various domain specific query languages.

---

<div class="post-metadata">

**Author:** ![datnamer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datnamer/32/3471_2.png) [@datnamer](https://discourse.julialang.org/u/datnamer)\
**Post date:** [March 27, 2019, 6:26pm UTC](https://discourse.julialang.org/t/ann-dataknots-jl-an-extensible-practical-and-coherent-algebra-of-query-combinators/22423/2 "2019-03-27T18:26:56Z")

</div>

This looks really cool! I’ll take a look.

Do you know offhand how it compares to @davidanthoff 's [Query.jl](https://github.com/queryverse/Query.jl)

---

<div class="post-metadata">

**Author:** ![leethargo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leethargo/32/6004_2.png) [@leethargo](https://discourse.julialang.org/u/leethargo)\
**Post date:** [March 27, 2019, 6:58pm UTC](https://discourse.julialang.org/t/ann-dataknots-jl-an-extensible-practical-and-coherent-algebra-of-query-combinators/22423/3 "2019-03-27T18:58:57Z")

</div>

Here are some points of comparison from what I’ve seen:

- Queries in DataKnots are built from subqueries. These can be assigned names and used independently, e.g., for testing or abstraction (like functions).
- DataKnots works on nested data, while Query works (mostly) with tabular data, row-by-row.
- DataKnots works with higher-order functions, while Query works with macros.

Also, for the first release, DataKnots works with its own datatype for in-memory columnar storage, while Query works on any datatype that implements the TableTraits(?) interface. But I’m guessing that this is going to change in future releases.

---

<div class="post-metadata">

**Author:** ![xitology](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xitology/32/7570_2.png) [@xitology](https://discourse.julialang.org/u/xitology)\
**Post date:** [March 27, 2019, 7:11pm UTC](https://discourse.julialang.org/t/ann-dataknots-jl-an-extensible-practical-and-coherent-algebra-of-query-combinators/22423/4 "2019-03-27T19:11:58Z")

</div>

> [@datnamer](#):
>
> Do you know offhand how it compares to @davidanthoff 's [Query.jl](https://github.com/queryverse/Query.jl)

It’s a great question! Let’s see…

- Both serve the same purpose, to query data.
- `Query.jl` is usable now, but I wouldn’t recommend `DataKnots.jl` to end users just yet. It does not provide an easy way to import data, and some basic combinators like group/sort/join are not yet implemented. We plan to add them soon…
- The biggest difference is the query model. Unlike `Query.jl`, `DataKnots.jl` is an algebra and does not rely on variables, binding operations, or macros. This gives it a distinct declarative flavor and makes it easier to construct queries incrementally or dynamically.

See also the old thread at [Differences with others query packages · Issue #1 · MechanicalRabbit/DataKnots.jl · GitHub](https://github.com/rbt-lang/DataKnots.jl/issues/1)

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [March 27, 2019, 8:22pm UTC](https://discourse.julialang.org/t/ann-dataknots-jl-an-extensible-practical-and-coherent-algebra-of-query-combinators/22423/5 "2019-03-27T20:22:53Z")

</div>

Just a few points:

- [Query.jl](https://github.com/queryverse/Query.jl) does _not_ work primarily on tabular data, it just happens to be the case that that is what folks use it for. In fact, the core of it doesn’t even know what a table is 🙂 The core abstraction in [Query.jl](https://github.com/queryverse/Query.jl) is that of iterators, not rows. Iterators of named tuples also work well, and those are tables, but that is really just a special case that happens to be popular.
- The macro part in [Query.jl](https://github.com/queryverse/Query.jl) is only providing syntactic sugar for 1) named tuples and 2) anonymous functions for the piped version of things (of course for the LINQ style queries it does more). The piped version of things conceptually is based on higher-order functions.
- One can use [Query.jl](https://github.com/queryverse/Query.jl) with hierarchical/nested data! There are lots of examples for that in the LINQ world. It might be less elegant than the DataKnots story, not sure.
- The core idea of [Query.jl](https://github.com/queryverse/Query.jl) is also based on monadic composition. There are lots of old Channel 9 videos that go into the details of that for the LINQ case, and pretty much everything said there applies to [Query.jl](https://github.com/queryverse/Query.jl) as well.
- “does not rely on variables, binding operations, or macros” I’m not entirely sure what you mean by this 🙂 But at least the macro part I think is not important, see above.

Great to see this effort, I really should look into the details a bit more!

---

<div class="post-metadata">

**Author:** ![xitology](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xitology/32/7570_2.png) [@xitology](https://discourse.julialang.org/u/xitology)\
**Post date:** [March 27, 2019, 10:04pm UTC](https://discourse.julialang.org/t/ann-dataknots-jl-an-extensible-practical-and-coherent-algebra-of-query-combinators/22423/6 "2019-03-27T22:04:25Z")

</div>

> [@davidanthoff](#):
>
> - The core idea of [Query.jl](https://github.com/queryverse/Query.jl) is also based on monadic composition. There are lots of old Channel 9 videos that go into the details of that for the LINQ case, and pretty much everything said there applies to [Query.jl](https://github.com/queryverse/Query.jl) as well.

LINQ and DataKnots are both based on monads, but they use different primitives as the building blocks, which give them very different flavors. LINQ uses monadic comprehension, which lets you transform a monadic container by a chain of monadic functions: MA \to (A \to MB) \to MB, then MB \to (B \to MC) \to MC. DataKnots leaves out containers and uses monadic composition (A \to MB) \to (B \to MC) \to (A \to MC). The difference is in the type of objects you want to deal with. With comprehension, you operate on monadic containers MX; with composition, on monadic functions X \to MY.

> [@](#):
>
> - “does not rely on variables, binding operations, or macros” I’m not entirely sure what you mean by this 🙂 But at least the macro part I think is not important, see above.

I agree that macros are not relevant. Regarding variables, this is just as I stated: DataKnots does not use bound variables. For example, compare a query `Filter(It.age .> 50)` with a similar LINQ expression `where p.age > 50`. LINQ example is only valid within a context of the `from` binder that binds `p`. On the other hand, `It.age .> 50` is a self-contained query and does not depend on any context.

---

<div class="post-metadata">

**Author:** ![cce](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cce/32/460_2.png) [@cce](https://discourse.julialang.org/u/cce)\
**Post date:** [March 27, 2019, 10:52pm UTC](https://discourse.julialang.org/t/ann-dataknots-jl-an-extensible-practical-and-coherent-algebra-of-query-combinators/22423/7 "2019-03-27T22:52:05Z")

</div>

Hello all. So, we’ve opted to release DataKnots v0.1.1 in an unfinished state, mostly by _pruning_ features so that we could demonstrate this query combinator approach, with working code and [documentation](https://rbt-lang.github.io/DataKnots.jl/v0.1.1/). Even so, there is quite a bit in this release:

- Scalars, vectors and tuples are constant queries (`Lift`).
- Composition (`>>`) and identity (`It`) lets one connect queries.
- Aggregates include `Count`, `Sum`, `Max` and `Min`.
- `Record` permits new structures to be constructed.
- `Filter` permits output rows to be tailored.
- `Drop` / `Take` allow results to be paged.
- `Keep` / `Given` permit intermediate results to be reused.
- Julia functions can be used as query combinators (`Lift`).

The ability to `Group`, `Join`, `Merge`, and `Sort` data, among many others, will be implemented in subsequent releases. Yet, as we add them, the storage system and query mechanics will gain additional complexity that will make the code less digestible. Hence, this particular release is a sweet-spot of functionality vs complexity that could be studied (and perhaps improved) by others.

Happy Hacking!

---

<div class="post-metadata">

**Author:** ![cce](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cce/32/460_2.png) [@cce](https://discourse.julialang.org/u/cce)\
**Post date:** [April 30, 2019, 11:35am UTC](https://discourse.julialang.org/t/ann-dataknots-jl-an-extensible-practical-and-coherent-algebra-of-query-combinators/22423/8 "2019-04-30T11:35:24Z")

</div>

DataKnots v0.5.0 is released. Compared to v0.1.0, this release includes:

- Added `@query` notation
- Added `Unique`, `Collect` and `Group` combinators
- Added `DataKnot(::Pair...)` constructor
- Added import/export using Tables.jl interface
- Improved rendering of oversized tables

We invite your feedback: [gitter](https://gitter.im/rbt-lang/rbt-proto), [github](https://github.com/rbt-lang/DataKnots.jl), and [docs](https://rbt-lang.github.io/DataKnots.jl/stable/).