I’ve recently been working with some fairly I/O-intensive workflows and found that DuckDB.jl tends to be the recommended option for larger-than-memory data processing.
That said, I don’t particularly enjoy writing SQL-like queries, while I do enjoy working with table-like pipelines. So, for no particularly good reason other than my own enjoyment while coding, I started looking for alternatives to DuckDB that support lazy/out-of-core data processing in Julia, but haven’t had much luck.
To the best of my knowledge, Polars.jl is the closest alternative, but it appears to be discontinued, with the only active fork still being highly experimental.
Does anyone know of other alternatives in the Julia ecosystem?
I feel your pain. I absolutely loathe writing SQL, and working with relational databases in general. Have a look at TidierDB.jl - I think it does what you want with the DuckDB backend.
I’ve been reviving Query.jl, and have a DuckDB backend now (GitHub - queryverse/QueryDuckDB.jl · GitHub). It is all very in flux right now, but if you like the Query surface syntax, this could be a way to make it work with large datasets.
I was unaware of the Tidier organization and it looks awesome in general. I was just skimming through TidierDB and it feels like something I will very much enjoy, thanks!
I was originally working with .csv files, however, I turned them into a .parquet file to allow for data streaming and save some storage space in the process. I have heard quite a bit about HDF5, but never really got to understand it, maybe now is the time!