# Larger than memory table format with lazy reads?

**URL:** <https://discourse.julialang.org/t/larger-than-memory-table-format-with-lazy-reads/45123>\
**Category:** General Usage\
**Tags:** array, dataframes, juliadb\
**Created:** [August 17, 2020, 6:10pm UTC](https://discourse.julialang.org/t/larger-than-memory-table-format-with-lazy-reads/45123 "2020-08-17T18:10:28Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [August 17, 2020, 6:10pm UTC](https://discourse.julialang.org/t/larger-than-memory-table-format-with-lazy-reads/45123/1 "2020-08-17T18:10:28Z")

</div>

Let’s say, I have some storage with a table-like data in it. For example, it is a single big binary file with serialized array of `StoredRow` structures:

```julia
struct StoredRow
    a::Float64
    b::Float64
    c::UInt16
end

```

Also, I have a function to read chunks of data from storage given a range of row index and convert them into array of StoredRow structures:  
`function readfromstorage(storage, range::UnitRange{Int64})::Array{StoredRow}`

If it was a small table, I could convert it into table (`DataFrame`, `StructArray`, or `IndexedTable`) and work with it directly:

```julia
# some small array:
vec = [StoredRow(rand(),rand(),i) for i = 1:100]

using DataFrames
df = DataFrame(vec)

using StructArrays, JuliaDB
s = StructArray(vec)
t = table(fieldarrays(s))

```

But how can I create a similar table object, if storage data does not fit in memory?  
Can I simply attach data source (or some other abstract interface type) to a table object, so it can lazy read data from source in chunks (and maybe cache the last readed results)?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [August 17, 2020, 6:26pm UTC](https://discourse.julialang.org/t/larger-than-memory-table-format-with-lazy-reads/45123/2 "2020-08-17T18:26:51Z")

</div>

This is a canonical use-case for JuliaDB.

---

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [August 17, 2020, 7:21pm UTC](https://discourse.julialang.org/t/larger-than-memory-table-format-with-lazy-reads/45123/3 "2020-08-17T19:21:14Z")

</div>

Can you please provide a minimal example with lazy-loading using JuliaDB?  
As I can see from its docs, there is `loadtable` only for CSV files, and no abstract datasource interfaces. And there is chunking based on distributed workers - how it can help in a task with one worker operating with table that does not fit into its RAM?  
Or maybe I miss something?

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [August 17, 2020, 7:40pm UTC](https://discourse.julialang.org/t/larger-than-memory-table-format-with-lazy-reads/45123/4 "2020-08-17T19:40:12Z")

</div>

I don’t know much about JuliaDB, sorry, only that this is one of it’s goals. Hopefully someone else can chime in and discuss whether or not I’m right about that.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [August 18, 2020, 1:23am UTC](https://discourse.julialang.org/t/larger-than-memory-table-format-with-lazy-reads/45123/5 "2020-08-18T01:23:20Z")

</div>

Seems like there is a need for this. I am not sure how maintained JuliaDB.jl is.

I am building something.

But perhaps DiskArray with DataFrames.jl will already work

---

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [August 19, 2020, 12:33pm UTC](https://discourse.julialang.org/t/larger-than-memory-table-format-with-lazy-reads/45123/6 "2020-08-19T12:33:09Z")

</div>

Thanks for pointing me to DiskArrays.jl! As I can see, there is abstract interface for chunked arrays, so I can implement it based on my own storage format. Then, in theory I can build DataFrame or IndexedTable from columns of custom array types.