I have to write a lot of objects into a large array and mutate them. The array will grow constantly and will eventually be too large for memory. The most convenient way to do this is Mmap but it seems that it is not possible to grow an mmaped array. Is there a way around this limitation?
AFAIK not. When you fill the length of the array, you can reallocate a new one and copy the contents.
Thanks, I trying some kind of
mutable struct HybridMmap
x.data = Mmap.mmap(x.data_io)
data_io = open(file_name, "a+")
x = HybridMmap(data_io, UInt8)
flush! every time an out of bounds access is needed.
So you have an array and a sink that allows you to append values at the end? This may work.
It is working, even though it is a little bit tedious. I should probably put the bounds checking and flushing into
I this a limitation of the mmap on the OS level or only of the implementation in Julia?
If you are asking about growing an
mmaped array, I think this is a built-in limitation of the
mmap interface as provided by the OS.