# Lazy database join
**URL:** https://discourse.julialang.org/t/lazy-database-join/37304
**Category:** General Usage
**Tags:** question, performance, dataframes, big-data
**Created:** [April 9, 2020, 11:55pm UTC](https://discourse.julialang.org/t/lazy-database-join/37304 "2020-04-09T23:55:47Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@e3c6](https://discourse.julialang.org/u/e3c6)
#### Post date: [April 9, 2020, 11:55pm UTC](https://discourse.julialang.org/t/lazy-database-join/37304/1 "2020-04-09T23:55:47Z")
I have a very large database `A`, and a much smaller one `B`. Since `A` is so large, I cannot load it in memory at once. I keep `A` as a gzip compressed CSV file on disk, and I want to load it lazily. On the other hand `B` is small and fits well into memory.
The goal is to perform a `join` between them, on a certain column shared by both `A` and `B`. See this example: [Joins · DataFrames.jl](https://juliadata.github.io/DataFrames.jl/stable/man/joins/).
How can I do this, without ever loading `A` fully into memory?
---
### Author:  [@tbeason](https://discourse.julialang.org/u/tbeason)
#### Post date: [April 10, 2020, 12:48am UTC](https://discourse.julialang.org/t/lazy-database-join/37304/2 "2020-04-10T00:48:16Z")
It’s possible that you could do this with JuliaDB although I admit I have no idea.
A different way could be to sort of “roll your own”. You can stream the larger file using `CSV.jl` (and `CodecZLib.jl` I suppose). See `CSV.Rows`. If you are only joining on one column, writing the join logic shouldn’t be too bad. Look at whatever key value the row of A that you read in has, stick the columns from A that you want onto the B table where the key exists.
---
### Author:  [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [April 10, 2020, 12:55am UTC](https://discourse.julialang.org/t/lazy-database-join/37304/3 "2020-04-10T00:55:07Z")
For a second I thought that you might be able to use [Query.jl](https://github.com/queryverse/Query.jl), but then I looked at the code again and joins are not streaming in that sense, unfortunately… Thinking about it, I think there is actually no reason why they couldn’t, I would just have to change the implementation. But not on the roadmap right now…
---
### Author:  [@Jakob](https://discourse.julialang.org/u/Jakob)
#### Post date: [April 10, 2020, 6:45am UTC](https://discourse.julialang.org/t/lazy-database-join/37304/4 "2020-04-10T06:45:35Z")
This is possible with JuliaDB, see the [docs](https://juliacomputing.github.io/JuliaDB.jl/latest/out_of_core/#Join-to-Big-Table-1). I’ve had some [problems](https://discourse.julialang.org/t/error-when-performing-join-on-distributed-tables-juliadb/35800) in the past with these joins and I’m not sure if JuliaDB is being actively maintained at the moment but often it also just worked fine so I’d just give it a try.
---
### Author:  [@lungben](https://discourse.julialang.org/u/lungben)
#### Post date: [April 10, 2020, 6:54am UTC](https://discourse.julialang.org/t/lazy-database-join/37304/5 "2020-04-10T06:54:05Z")
An alternative would be to use a database like PostgreSQL.
It can be easily set up using Docker and interplays with Julia and DataFrames.jl using LibPQ.jl.
However, this may be overkill for your use case…