Growing mmaped arrays

General Usage
1

I have to write a lot of objects into a large array and mutate them. The array will grow constantly and will eventually be too large for memory. The most convenient way to do this is Mmap but it seems that it is not possible to grow an mmaped array. Is there a way around this limitation?

2

AFAIK not. When you fill the length of the array, you can reallocate a new one and copy the contents.

3

Thanks, I trying some kind of IOStream/Mmap hybrid:

mutable struct HybridMmap
  data_io::IOStream
  data::Array{UInt8}
end

function flush!(x::HybridMmap)
  flush(x.data_io)
  seek(x.data_io, 0)
  x.data = Mmap.mmap(x.data_io)
  seekend(x.data_io)
  return x
end

function HybridMmap(file_name)
  data_io  = open(file_name, "a+")
  x = HybridMmap(data_io, UInt8[])
  flush!(x)
end

and then flush! every time an out of bounds access is needed.

4

So you have an array and a sink that allows you to append values at the end? This may work.

5

It is working, even though it is a little bit tedious. I should probably put the bounds checking and flushing into getindex.

1 Like
6

I this a limitation of the mmap on the OS level or only of the implementation in Julia?

7

If you are asking about growing an mmaped array, I think this is a built-in limitation of the mmap interface as provided by the OS.