# Is there a DataFrame that can be memory mapped to a file?
**URL:** https://discourse.julialang.org/t/is-there-a-dataframe-that-can-be-memory-mapped-to-a-file/96926
**Category:** Data
**Tags:** question, dataframes, mmap
**Created:** [April 1, 2023, 4:56am UTC](https://discourse.julialang.org/t/is-there-a-dataframe-that-can-be-memory-mapped-to-a-file/96926 "2023-04-01T04:56:35Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@ktdq](https://discourse.julialang.org/u/ktdq)
#### Post date: [April 1, 2023, 4:56am UTC](https://discourse.julialang.org/t/is-there-a-dataframe-that-can-be-memory-mapped-to-a-file/96926/1 "2023-04-01T04:56:35Z")
Just saw [Make loading weights 10-100x faster](https://github.com/ggerganov/llama.cpp/pull/613) on [Hacker News](https://news.ycombinator.com/item?id=35393284)
Is there something equivalent in Julia? Is there a way to store DataFrame in a file so it loads instantly via mmap?
---
### Author:  [@ktdq](https://discourse.julialang.org/u/ktdq)
#### Post date: [April 1, 2023, 6:39am UTC](https://discourse.julialang.org/t/is-there-a-dataframe-that-can-be-memory-mapped-to-a-file/96926/2 "2023-04-01T06:39:24Z")
ChatGPT suggests the following:
```julia
for (name, T) in zip(colnames, coltypes)
data = Mmap.mmap(io, Vector{T}, (nrows,))
df[!, name] = data
seek(io, position(io) + sizeof(T) * nrows)
end
```
---
### Author:  [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)
#### Post date: [April 1, 2023, 10:44am UTC](https://discourse.julialang.org/t/is-there-a-dataframe-that-can-be-memory-mapped-to-a-file/96926/3 "2023-04-01T10:44:12Z")
Fyi, this is covered in subchapter 8.4 of @bkamins’s [julia-for-data-analysis-book](https://discourse.julialang.org/t/julia-for-data-analysis-book/78090), where it is shown how data frames can be stored in different formats, including Apache arrow.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [April 1, 2023, 10:45am UTC](https://discourse.julialang.org/t/is-there-a-dataframe-that-can-be-memory-mapped-to-a-file/96926/4 "2023-04-01T10:45:54Z")
Yes, using Arrow.jl is a standard way to do it AFAICT.
---
### Author:  [@Impressium](https://discourse.julialang.org/u/Impressium)
#### Post date: [December 12, 2023, 11:31pm UTC](https://discourse.julialang.org/t/is-there-a-dataframe-that-can-be-memory-mapped-to-a-file/96926/5 "2023-12-12T23:31:22Z")
But is there a way to update the dataframe using Arrow.jl. I would like to `push!` new rows to it.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [December 13, 2023, 7:09am UTC](https://discourse.julialang.org/t/is-there-a-dataframe-that-can-be-memory-mapped-to-a-file/96926/6 "2023-12-13T07:09:09Z")
Arrow.jl is read only.
---
### Author:  [@Impressium](https://discourse.julialang.org/u/Impressium)
#### Post date: [December 13, 2023, 7:23am UTC](https://discourse.julialang.org/t/is-there-a-dataframe-that-can-be-memory-mapped-to-a-file/96926/7 "2023-12-13T07:23:47Z")
But is there a way to load huge amounts of data and update & add more data?
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [December 13, 2023, 10:35am UTC](https://discourse.julialang.org/t/is-there-a-dataframe-that-can-be-memory-mapped-to-a-file/96926/8 "2023-12-13T10:35:23Z")
You would need to load it by chunks AFAICT