# Read-only memory-mapped files

**URL:** https://discourse.julialang.org/t/read-only-memory-mapped-files/12098
**Category:** Internals & Design
**Tags:** data
**Created:** [July 2, 2018, 5:04pm UTC](https://discourse.julialang.org/t/read-only-memory-mapped-files/12098 "2018-07-02T17:04:09Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![dmbates](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmbates/32/44_2.png) [@dmbates](https://discourse.julialang.org/u/dmbates)
#### Post date: [July 2, 2018, 5:04pm UTC](https://discourse.julialang.org/t/read-only-memory-mapped-files/12098/1 "2018-07-02T17:04:09Z")

</div>

I am developing some code in [`BEDFIles.jl`](https://github.com/dmbates/BEDFiles.jl.git) that may be used with very large binary data files. The data are accessed as a read-only memory-mapped `Matrix{UInt8}` using column-oriented algorithms whenever possible.

As I understand it, there shouldn’t be a problem with having a very large file if I am only accessing a small set of adjacent columns. Suppose that I have 100,000 rows and 10 million columns but I only access the first 10,000 columns. I believe that the columns beyond 10,000 will never need to appear in memory - that they are essentially held as a kind of a promise by the operating system (which would be Linux - I don’t care if Windows does dumb things with memory-mapped files). Is this correct?

---

<div class="post-metadata">

### Author: ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [July 2, 2018, 5:33pm UTC](https://discourse.julialang.org/t/read-only-memory-mapped-files/12098/2 "2018-07-02T17:33:55Z")

</div>

If you are opening a memory mapped file, yes that is correct.

Admittedly I am still rather hazy on some of the details, but you can get a partial description [here](https://en.wikipedia.org/wiki/Memory-mapped_I/O).

It should go without saying that you still have to be careful about actually copying data out of the memory mapped array.

---

<div class="post-metadata">

### Author: ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [July 3, 2018, 7:08am UTC](https://discourse.julialang.org/t/read-only-memory-mapped-files/12098/3 "2018-07-03T07:08:36Z")

</div>

Yes, your expectations are correct. I have `mmap`ped 500GB files on a 16GB machine without any problems, the OS (in my case, Linux) takes care of the memory operations very transparently, paging on demand.

For 10^5\cdot10^4=10^9 `UInt8`s, that’s 1GB, so chances are the whole section could just fit in memory, making access really fast.
