# Memory usage with mmap file

**URL:** <https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883>\
**Category:** Performance\
**Tags:** binaryio, garbage-collection, mmap\
**Created:** [April 23, 2021, 2:57pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883 "2021-04-23T14:57:03Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![KKesgin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kkesgin/32/32168_2.png) [@KKesgin](https://discourse.julialang.org/u/KKesgin)\
**Post date:** [April 23, 2021, 2:57pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/1 "2021-04-23T14:57:03Z")

</div>

Hi All  
I’m accessing a very large binary file randomly on windows. I don’t dare to put in all the code but I managed to replicate the behavior with a small loop. Could someone please explain to me why this is causing infinite memory growth (or at least an unexpectedly large growth, despite using mmap) and how I can circumvent it?

using Mmap

io = open(“path.bin”, “r”)  
rawDataMap = Mmap.mmap(io, Matrix{Int16}, (385, 88173048),grow=false,shared=false)  
close(io)

count = [1]  
while true  
println(sum(rawDataMap[:,count[1]:count[1]+75]))  
count[1] += rand(75:100)  
end

I would expect the loop to be garbage collected but apparently not? And no Gc.gc() doesn’t help.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [April 23, 2021, 4:00pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/2 "2021-04-23T16:00:03Z")

</div>

Julia doesn’t have any control over how the memory pages backed by an mmapped array are managed — that is entirely up to the operations system. The behavior you’re seeing is what I’d expect the OS to do: lazily adding actual backing memory pages to the mmapped region as you access it and then not paging them out until memory pressure gets so great that your system is completely out of memory (and maybe not even then depending on the OS and how it’s configured).

---

<div class="post-metadata">

**Author:** ![KKesgin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kkesgin/32/32168_2.png) [@KKesgin](https://discourse.julialang.org/u/KKesgin)\
**Post date:** [April 23, 2021, 4:26pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/3 "2021-04-23T16:26:24Z")

</div>

I appreciate the response, however that means that there is no alternative here to work on files larger than memory size + leftover storage space?

I basically have a 400 GB file with events occurring at “random” times.  
And I have a list containing said event times.  
I want to process (not just view) a small amount of data points surrounding each event time on that list by looking at original 400 gb binary file.  
How does one work this out with reasonable computing resources (32 GB RAM, 128 gb free storage space) if mmap isn’t the answer?

---

<div class="post-metadata">

**Author:** ![mgkuhn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mgkuhn/32/6276_2.png) [@mgkuhn](https://discourse.julialang.org/u/mgkuhn)\
**Post date:** [April 23, 2021, 4:58pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/4 "2021-04-23T16:58:34Z")

</div>

Try

```julia
view(rawDataMap, :, count[1]:count[1]+75)

```

instead of

```julia
rawDataMap[:,count[1]:count[1]+75]

```

to avoid copying data from the memory-mapped file blocks into your process memory.

---

<div class="post-metadata">

**Author:** ![KKesgin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kkesgin/32/32168_2.png) [@KKesgin](https://discourse.julialang.org/u/KKesgin)\
**Post date:** [April 23, 2021, 5:00pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/5 "2021-04-23T17:00:28Z")

</div>

Thanks for your response, however I need to process said ranges.  
Let’s say for the sake of simplicity I want to:

println(sum(rawDataMap[:,count[1]:count[1]+75]))

This still causes memory to explode (with views too)

---

<div class="post-metadata">

**Author:** ![mgkuhn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mgkuhn/32/6276_2.png) [@mgkuhn](https://discourse.julialang.org/u/mgkuhn)\
**Post date:** [April 23, 2021, 5:04pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/6 "2021-04-23T17:04:44Z")

</div>

If you need a copy anyway, `mmap` probably has little advantage over `seek` and `read!`.

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [April 23, 2021, 5:07pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/7 "2021-04-23T17:07:28Z")

</div>

Can’t you work with the file? I mean, do not create the `mmapped` array but instead use `seek` over the `io` object to get the exact ranges you want?

---

<div class="post-metadata">

**Author:** ![KKesgin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kkesgin/32/32168_2.png) [@KKesgin](https://discourse.julialang.org/u/KKesgin)\
**Post date:** [April 23, 2021, 5:09pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/8 "2021-04-23T17:09:42Z")

</div>

If I’m not mistaken “read” isn’t really a possibility if I want to read 10 kbs of data midway through 400 gb file. But I appreciate for pointing out “seek” exists. I will try that and update the response as solution if that works out as soon as possible.

---

<div class="post-metadata">

**Author:** ![zgornel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zgornel/32/217487_2.png) [@zgornel](https://discourse.julialang.org/u/zgornel)\
**Post date:** [April 23, 2021, 5:09pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/9 "2021-04-23T17:09:44Z")

</div>

What if you do:

```julia
while true
   tmp = rawDataMap[:,count[1]:count[1]+75]
   f(tmp)
   tmp=[] # explicit release
   GC.gc()
end

```

This works (under Linux at least) pretty well

---

<div class="post-metadata">

**Author:** ![KKesgin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kkesgin/32/32168_2.png) [@KKesgin](https://discourse.julialang.org/u/KKesgin)\
**Post date:** [April 23, 2021, 5:11pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/10 "2021-04-23T17:11:02Z")

</div>

Thanks for the response. I’m afraid that doesn’t work on Windows. That was one of the first things I tried.

---

<div class="post-metadata">

**Author:** ![mgkuhn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mgkuhn/32/6276_2.png) [@mgkuhn](https://discourse.julialang.org/u/mgkuhn)\
**Post date:** [April 23, 2021, 5:12pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/11 "2021-04-23T17:12:47Z")

</div>

Try `seek` with [`read!`](https://docs.julialang.org/en/v1/base/io-network/#Base.read!), which takes an array to be filled as an argument.

---

<div class="post-metadata">

**Author:** ![mgkuhn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mgkuhn/32/6276_2.png) [@mgkuhn](https://discourse.julialang.org/u/mgkuhn)\
**Post date:** [April 23, 2021, 5:18pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/12 "2021-04-23T17:18:29Z")

</div>

I have no practical experience with how mmap() behaves on Windows, and would be curious if e.g. Linux has the same problem.

You want to find out if the memory you run out of is

- allocated by Julia’s memory manager (and thus could be garbage collected by Julia)
- allocated to the Julia process but not part of the heap memory that is managed by Julia’s memory manager (and thus `gc()` would have no effect),
- is outside the Julia process and instead part of the operating-system’s block-buffer cache (which all processes share), and the problem then may be that OS just doesn’t deal well with that situation.

Is your OS swap space big enough for the file? See Increasing [Virtual Memory in Windows 10](https://www.x-plane.com/kb/increasing-virtual-memory-on-windows/)

---

<div class="post-metadata">

**Author:** ![mgkuhn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mgkuhn/32/6276_2.png) [@mgkuhn](https://discourse.julialang.org/u/mgkuhn)\
**Post date:** [April 23, 2021, 5:28pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/13 "2021-04-23T17:28:29Z")

</div>

I hadn’t realized that you can use an `mmap()`ed file after closing the corresponding file descriptor. If Julia really does not `munmap()` the mapping when you close the file, when does Julia then undo the memory map?

The [documentation](https://docs.julialang.org/en/v1/stdlib/Mmap/) doesn’t say anything on how unmapping a memory-mapped file works.

---

<div class="post-metadata">

**Author:** ![KKesgin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kkesgin/32/32168_2.png) [@KKesgin](https://discourse.julialang.org/u/KKesgin)\
**Post date:** [April 23, 2021, 5:30pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/14 "2021-04-23T17:30:42Z")

</div>

I share the same confusion. Understanding that behavior could have helped identifying the problem. I’m afraid documentation page really isn’t helpful at all.

---

<div class="post-metadata">

**Author:** ![zgornel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/zgornel/32/217487_2.png) [@zgornel](https://discourse.julialang.org/u/zgornel)\
**Post date:** [April 23, 2021, 5:48pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/15 "2021-04-23T17:48:54Z")

</div>

I think a memory mapped file should be looked more like virtual memory associated to a process.

Once a file is mapped, it becomes memory that is accessible to the `julia` process at some address. In Linux, one can check by running `cat /proc/<julia pid>/maps` and it will turn out that some memory region is mapped to the file - this stuff is handled transparently by the OS (address transformations etc)

Consequently, the mapped file can be an actual `Matrix` object in `julia` whose values happen to be not in RAM but disk.

---

<div class="post-metadata">

**Author:** ![mgkuhn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mgkuhn/32/6276_2.png) [@mgkuhn](https://discourse.julialang.org/u/mgkuhn)\
**Post date:** [April 23, 2021, 7:09pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/16 "2021-04-23T19:09:58Z")

</div>

BTW, looking at [the source code](https://github.com/JuliaLang/julia/blob/6aaedecc447e3d8226d5027fb13d0c3cbfbfea2a/stdlib/Mmap/src/Mmap.jl#L184-L238), [mmap()](https://docs.julialang.org/en/v1/stdlib/Mmap/#Mmap.mmap) registers a [finalizer](https://docs.julialang.org/en/v1/base/base/#Base.finalizer) for the array, such that when the array is garbage collected, the memory mapping will be removed. Makes sense.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [April 23, 2021, 7:26pm UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/17 "2021-04-23T19:26:19Z")

</div>

I don’t get why seek and read wouldn’t work. Mmap seems like the wrong tool here: you are essentially telling the kernel “I want all of this file in my process memory (but do it lazily in case I don’t actually need to touch all of it).” Using seek and read, on the other hand is precisely the right way to read only a small part of a large file when you don’t want the rest of it in memory. In particular, if you want to process a large file “in streaming fashion”, you want to use read rather than mmap.

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [September 16, 2021, 3:38am UTC](https://discourse.julialang.org/t/memory-usage-with-mmap-file/59883/18 "2021-09-16T03:38:55Z")

</div>

> [@StefanKarpinski](#):
>
> Using seek and read, on the other hand is precisely the right way to read only a small part of a large file when you don’t want the rest of it in memory.

Is there a package that implements an `AbstractArray` backed by “seek and read” of a file? I know `Arrow` does this, but it comes with a lot of baggage, and only supports vectors AFAICT…