[ANN,RFC] DBCollections.jl – use Julia data manipulation functions for databases

[aplavin]

@xiaodai

I think the impression I get is that usability and understandability is not a priority atm. So it immediately signals to me as something I may not want to invest time in until those become priorities.

Totally, that’s fair! For now, DBCollections is an experiment: can we just use existing Julia data manipulation functions for working with SQL databases?
Turns out the answer is “yes”, as DBCollections demonstrate. That’s why this thread!
But it’s definitely not a fleshed out implementation, needless to say about sparse documentation.

I’ve been using DBCollections for DuckDB and MySQL databases during the past weeks, and find it convenient and useful. Doesn’t mean it will be useful for you specifically!

@xiaodai

The first example already lost me, what’s @p what’s @o? A simple description would go a long way.

Sorry, I jumped straight into what DBCollections does in the first post, and didn’t explain the functions/macros from other packages. The reason is they aren’t specific to databases and defined elsewhere: @p in DataPipes.jl, @o in Accessors.jl.

One can use DBCollections without this syntactic sugar, it’s just less convenient. For example, filter(Base.Fix2(>, 0) ∘ PropertyLens(:a), data) works whenever data is a regular Julia collection of elements: it selects those x with x.a > 0. With DBCollections, the same code works for data being an SQL table!
@o simply lets you write the inner function in a more readable way: filter((@o _.a > 0), data).
And @p gets you composable piping, @p data |> filter(@o _.a > 0) |> ....

Again, both aren’t directly related to DBCollections, just convenient to be used together.

[liuyxpp]

I am also very confused by the @p and @o. I thought they come from DBCollections.jl. But it seems I am wrong. It is a bad idea to show code that depends on other deps without any explanations, especially for those macros since they kind of doing some magic things.

[aplavin]

The idea was to demonstrate:

1. Here is existing code that already works for regular Julia collections (unrelated to DBCollections)
2. We want to use the exact same code for SQL databases
3. Solution: just pass a DBCollection instead of a Julia collection!

Ie, I don’t design any new API in this thread, only add SQL support to existing data manipulation functions.

Judging by the comments, this didn’t really come clearly at first. Sorry! Hope now it’s more understandable with these explanations (:
Feel free to suggest better wording for documentation.