Alternatives for lazy/out-of-core data processing in Julia

General Usage
1

Hi all,

I’ve recently been working with some fairly I/O-intensive workflows and found that DuckDB.jl tends to be the recommended option for larger-than-memory data processing.

That said, I don’t particularly enjoy writing SQL-like queries, while I do enjoy working with table-like pipelines. So, for no particularly good reason other than my own enjoyment while coding, I started looking for alternatives to DuckDB that support lazy/out-of-core data processing in Julia, but haven’t had much luck.

To the best of my knowledge, Polars.jl is the closest alternative, but it appears to be discontinued, with the only active fork still being highly experimental.

Does anyone know of other alternatives in the Julia ecosystem?

Best,
P.

2

I feel your pain. I absolutely loathe writing SQL, and working with relational databases in general. Have a look at TidierDB.jl - I think it does what you want with the DuckDB backend.

3

You could try DuckDB with FunSQL. I don’t know SQL very well, but use it occasionally. This section of its manual convinced me to use it.