I have a very large database A, and a much smaller one B. Since A is so large, I cannot load it in memory at once. I keep A as a gzip compressed CSV file on disk, and I want to load it lazily. On the other hand B is small and fits well into memory. The goal is to perform a join between them, on a c…

It’s possible that you could do this with JuliaDB although I admit I have no idea. A different way could be to sort of “roll your own”. You can stream the larger file using CSV.jl (and CodecZLib.jl I suppose). See CSV.Rows. If you are only joining on one column, writing the join logic shouldn’t be …

For a second I thought that you might be able to use Query.jl , but then I looked at the code again and joins are not streaming in that sense, unfortunately… Thinking about it, I think there is actually no reason why they couldn’t, I would just have to change the implementation. But not on the roadma…

An alternative would be to use a database like PostgreSQL. It can be easily set up using Docker and interplays with Julia and DataFrames.jl using LibPQ.jl. However, this may be overkill for your use case…

Lazy database join

General Usage

Jakob April 10, 2020, 6:45am 4

This is possible with JuliaDB, see the docs. I’ve had some problems in the past with these joins and I’m not sure if JuliaDB is being actively maintained at the moment but often it also just worked fine so I’d just give it a try.

Topic		Replies	Views
Best practices for local exploratory data analysis Data	15	1884	October 16, 2024
Larger than memory table format with lazy reads? General Usage array , dataframes , juliadb	5	1343	August 19, 2020
Using JuliaDB to create larger than memory datasets and work with them? General Usage	3	1155	October 15, 2019
Moving to juliadb from dataframes and getting error joining tables, dataframes join ok General Usage juliadb	1	376	July 8, 2022
Alternatives for lazy/out-of-core data processing in Julia General Usage package	9	455	September 25, 2026

Lazy database join

Related topics