# Growing mmaped arrays

**URL:** <https://discourse.julialang.org/t/growing-mmaped-arrays/18709>\
**Category:** General Usage\
**Created:** [December 16, 2018, 9:48am UTC](https://discourse.julialang.org/t/growing-mmaped-arrays/18709 "2018-12-16T09:48:07Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![gdkrmr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdkrmr/32/2791_2.png) [@gdkrmr](https://discourse.julialang.org/u/gdkrmr)\
**Post date:** [December 16, 2018, 9:48am UTC](https://discourse.julialang.org/t/growing-mmaped-arrays/18709/1 "2018-12-16T09:48:07Z")

</div>

I have to write a lot of objects into a large array and mutate them. The array will grow constantly and will eventually be too large for memory. The most convenient way to do this is Mmap but it seems that it is not possible to grow an mmaped array. Is there a way around this limitation?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 16, 2018, 10:08am UTC](https://discourse.julialang.org/t/growing-mmaped-arrays/18709/2 "2018-12-16T10:08:11Z")

</div>

AFAIK not. When you fill the length of the array, you can reallocate a new one and copy the contents.

---

<div class="post-metadata">

**Author:** ![gdkrmr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdkrmr/32/2791_2.png) [@gdkrmr](https://discourse.julialang.org/u/gdkrmr)\
**Post date:** [December 16, 2018, 10:53am UTC](https://discourse.julialang.org/t/growing-mmaped-arrays/18709/3 "2018-12-16T10:53:39Z")

</div>

Thanks, I trying some kind of `IOStream`/`Mmap` hybrid:

```julia
mutable struct HybridMmap
  data_io::IOStream
  data::Array{UInt8}
end

function flush!(x::HybridMmap)
  flush(x.data_io)
  seek(x.data_io, 0)
  x.data = Mmap.mmap(x.data_io)
  seekend(x.data_io)
  return x
end

function HybridMmap(file_name)
  data_io = open(file_name, "a+")
  x = HybridMmap(data_io, UInt8[])
  flush!(x)
end

```

and then `flush!` every time an out of bounds access is needed.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 16, 2018, 12:22pm UTC](https://discourse.julialang.org/t/growing-mmaped-arrays/18709/4 "2018-12-16T12:22:52Z")

</div>

So you have an array and a sink that allows you to append values at the end? This may work.

---

<div class="post-metadata">

**Author:** ![gdkrmr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdkrmr/32/2791_2.png) [@gdkrmr](https://discourse.julialang.org/u/gdkrmr)\
**Post date:** [December 16, 2018, 5:45pm UTC](https://discourse.julialang.org/t/growing-mmaped-arrays/18709/5 "2018-12-16T17:45:18Z")

</div>

It is working, even though it is a little bit tedious. I should probably put the bounds checking and flushing into `getindex`.

---

<div class="post-metadata">

**Author:** ![gdkrmr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdkrmr/32/2791_2.png) [@gdkrmr](https://discourse.julialang.org/u/gdkrmr)\
**Post date:** [December 16, 2018, 8:19pm UTC](https://discourse.julialang.org/t/growing-mmaped-arrays/18709/6 "2018-12-16T20:19:58Z")

</div>

I this a limitation of the mmap on the OS level or only of the implementation in Julia?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 17, 2018, 6:47am UTC](https://discourse.julialang.org/t/growing-mmaped-arrays/18709/7 "2018-12-17T06:47:38Z")

</div>

If you are asking about growing an `mmap`ed array, I think this is a built-in limitation of the `mmap` interface as provided by the OS.