[ANN] SQLCollections.jl – use Julia data manipulation functions for databases

aplavin
Tue, 10 Sep 2024 09:57:36 +0000
Package Announcements

Imagine if you could use the exact same code to manipulate regular Julia collections and SQL tables… SQLCollections.jl is the package to achieve that!

There is no shortage of Julia packages that provide convenient interface to querying databases – see a comparison in the README.
SQLCollections.jl is unique in that it allows you reuse the same code, the same functions without any modification for both in-memory collections (arrays, tables) and databases.
It turned out to be quite simple to implement, with just ~250 LOC without any macro definitions!

SQLCollections doesn’t define any macros, the interface is fundamentally function-based. No special parsing/interpolation rules to remember, less implicit magic overall!
This makes SQLCollections.jl play nicely with convenience macros from other packages. Most useful are Accessors and DataPipes.jl.

Some simple examples – here, data can either be a regular Julia collection or an SQLCollection.

using SQLCollections, Accessors

data = SQLCollection(mydb, :tablename)

# the actual SQLCollections interface:
# use familiar functions like map/filter, but need to pass inspectable function objects – not anonymous functions like x -> x.a > 0
filter(Base.Fix2(>, 0) ∘ PropertyLens(:a), data)

# of course, writing nontrivial functions this way is not too convenient
# luckily, the Accessors.@o macro provides a nice alternative:
filter((@o _.a > 0), data)

using DataPipes, AccessorsExtra
# for multi-step data manipulation pipelines, the @p macro from DataPipes is a natural fit:
@p let
    data
    filter(@o _.height > 180 && _.weight < 80)
    map(@o (name=_.name, ratio=_.weight / _.height))
    collect  # optional for Julia collections, required for SQLCollections – returns a StructArray for them
end

# grouping and other functions are supported:
@p let
    data
    map(@o (i=_.i, b=round(_.i/4.5)))
    group_vg(@o (a=_.i / 3.5,))
    map(@o (a=key(_).a, avg=mean(_.b), cnt=length(_)))
    collect
end

More examples and coming, see tests for now.

SQLCollections.jl uses FunSQL.jl under the hood, and works with any database supported by it (SQLite, DuckDB, Postgres, MySQL, …).
A large set of operations is already supported by DBCollections:

map/filter/sort/Iterators.drop/first/…

group (from FlexiGroups.jl)

push!/append!/copy!

Coming soon:

join (from FlexiJoins.jl)

nested structures support, translated to JSON operations in SQL

32 posts - 9 participants