Hi all, I’ve recently been working with some fairly I/O-intensive workflows and found that DuckDB.jl tends to be the recommended option for larger-than-memory data processing. That said, I don’t particularly enjoy writing SQL-like queries, while I do enjoy working with table-like pipelines. So, fo…

I feel your pain. I absolutely loathe writing SQL, and working with relational databases in general. Have a look at TidierDB.jl - I think it does what you want with the DuckDB backend.

You could try DuckDB with FunSQL. I don’t know SQL very well, but use it occasionally. This section of its manual convinced me to use it.

What kind of data do you have? Would Zarr.jl or HDF5.jl be appropriate?

I’ve been reviving Query.jl, and have a DuckDB backend now ( GitHub - queryverse/QueryDuckDB.jl · GitHub ). It is all very in flux right now, but if you like the Query surface syntax, this could be a way to make it work with large datasets.

Thread [image] Reading large CSV files New to Julia Hi, is there any way to read a 88 GB csv file into a DataFrame? It kills julia, even with the -t auto option. Thanks! has a related discussion and advice.

Many thanks for the recommendation! I was unaware of the Tidier organization and it looks awesome in general. I was just skimming through TidierDB and it feels like something I will very much enjoy, thanks!

I was originally working with .csv files, however, I turned them into a .parquet file to allow for data streaming and save some storage space in the process. I have heard quite a bit about HDF5, but never really got to understand it, maybe now is the time!

If you are dealing with tabular data, you may be interested in HDF5: HDF5 Table Specification Version 1.0 Standardizing Tabular Data: Introducing the HDF5 Extension Proposal HEP001 -... In this episode of “Call the Doctor,” Aleksandar Jelenak expl…

A few other options: If your data is spread over multiple files, FileTrees.jl might be a good fit. Arrow.jl mmaps (as long as you dont use compression) so you can have a lazy on disk DataFrame.

Alternatives for lazy/out-of-core data processing in Julia

General Usage

stephancb September 24, 2026, 11:25am 6

Thread

has a related discussion and advice.

Topic		Replies	Views
What's the latest and greatest in data in Julia Data	29	2909	August 15, 2024
A serious data start-up structured around a Julia data manipulation framework for larger-than-RAM data Offtopic	24	1287	September 16, 2024
JuliaData BoF @ JuliaCon2023 discussion Data discussion	2	543	August 14, 2023
ANN: JuliaDB.jl Community	37	10565	May 11, 2017
Package for reading/writing ~100GB data files General Usage	10	3096	November 17, 2018

Alternatives for lazy/out-of-core data processing in Julia

Related topics