# Storage-based array that implements @views?

**URL:** <https://discourse.julialang.org/t/storage-based-array-that-implements-views/52640>\
**Category:** General Usage\
**Tags:** question, array, memory-allocation\
**Created:** [December 31, 2020, 1:47am UTC](https://discourse.julialang.org/t/storage-based-array-that-implements-views/52640 "2020-12-31T01:47:05Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![mkarikom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkarikom/32/7096_2.png) [@mkarikom](https://discourse.julialang.org/u/mkarikom)\
**Post date:** [December 31, 2020, 1:47am UTC](https://discourse.julialang.org/t/storage-based-array-that-implements-views/52640/1 "2020-12-31T01:47:05Z")

</div>

I would like to do a piece-wise reduction (on sub-arrays) of a large array that is too big to fit into memory. Is there a ‘storage-based array’ that implements @views?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [December 31, 2020, 2:00am UTC](https://discourse.julialang.org/t/storage-based-array-that-implements-views/52640/2 "2020-12-31T02:00:25Z")

</div>

Are you looking for [https://github.com/JuliaArrays/MappedArrays.jl](https://github.com/JuliaArrays/MappedArrays.jl)?

---

<div class="post-metadata">

**Author:** ![mkarikom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkarikom/32/7096_2.png) [@mkarikom](https://discourse.julialang.org/u/mkarikom)\
**Post date:** [December 31, 2020, 2:07am UTC](https://discourse.julialang.org/t/storage-based-array-that-implements-views/52640/3 "2020-12-31T02:07:06Z")

</div>

Thanks @Oscar_Smith, I’m not really looking for implicit mapping but rather a data structure whose allocation size is limited by storage (not memory) that I can get [sub-arrays of](https://docs.julialang.org/en/v1/devdocs/subarrays/#Index-replacement). Basically this would allow me to port a lot of existing code to handle some new data.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 31, 2020, 2:21am UTC](https://discourse.julialang.org/t/storage-based-array-that-implements-views/52640/4 "2020-12-31T02:21:35Z")

</div>

The Mmap package provides memory-mapped file-based arrays that sound like they should do what you want. Only the portion you are accessing is paged into memory.

(Every AbstractArray subtype supports `@views` and subarrays.)

---

<div class="post-metadata">

**Author:** ![mkarikom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkarikom/32/7096_2.png) [@mkarikom](https://discourse.julialang.org/u/mkarikom)\
**Post date:** [December 31, 2020, 9:13pm UTC](https://discourse.julialang.org/t/storage-based-array-that-implements-views/52640/5 "2020-12-31T21:13:15Z")

</div>

Thanks, @stevengj. Even better that it’s in the standard library.

---

<div class="post-metadata">

**Author:** ![mkarikom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkarikom/32/7096_2.png) [@mkarikom](https://discourse.julialang.org/u/mkarikom)\
**Post date:** [December 31, 2020, 9:47pm UTC](https://discourse.julialang.org/t/storage-based-array-that-implements-views/52640/6 "2020-12-31T21:47:34Z")

</div>

Hi, @stevengj. For benchmarking the Mmap-based implementation, what is the best way to create some files with large random arrays?

The following (adapted from the [docs](https://docs.julialang.org/en/v1/stdlib/Mmap/#Memory-mapped-I/O)) obviously doesn’t work because it has to create and write the whole array at once:

```julia
n=10
s = open("/tmp/mmap.bin", "w+")
write(s, 10)
write(s, 10^n)
write(s, rand(10, 10^n))

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 31, 2020, 10:28pm UTC](https://discourse.julialang.org/t/storage-based-array-that-implements-views/52640/7 "2020-12-31T22:28:39Z")

</div>

You could just call `write` in a loop…, e.g.

```julia
open("mmap.bin", "w") do io
    write(io, 10^10)
    for i = 1:1000
        write(io, rand(10^7))
    end
end

```

---

<div class="post-metadata">

**Author:** ![mkarikom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkarikom/32/7096_2.png) [@mkarikom](https://discourse.julialang.org/u/mkarikom)\
**Post date:** [December 31, 2020, 10:43pm UTC](https://discourse.julialang.org/t/storage-based-array-that-implements-views/52640/8 "2020-12-31T22:43:29Z")

</div>

Thanks!  
Is the `do` construct required here? I used almost the same for-loop strategy [in this post](https://discourse.julialang.org/t/how-to-stream-a-large-random-to-a-file/52673) but if fails.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 31, 2020, 10:56pm UTC](https://discourse.julialang.org/t/storage-based-array-that-implements-views/52640/9 "2020-12-31T22:56:50Z")

</div>

> [@mkarikom](#):
>
> Is the `do` construct required here?

No, but it’s good style when writing a file, since it automatically closes the file at the end of the `do...end` even if an exception occurs. It’s equivalent to

```julia
io = open(...)
try
    ...write stuff...
finally
    close(io)
end

```

but is less verbose.

---

<div class="post-metadata">

**Author:** ![mkarikom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkarikom/32/7096_2.png) [@mkarikom](https://discourse.julialang.org/u/mkarikom)\
**Post date:** [January 1, 2021, 7:41pm UTC](https://discourse.julialang.org/t/storage-based-array-that-implements-views/52640/10 "2021-01-01T19:41:04Z")

</div>

Thanks, @stevengj!

Also I realized that the `w+` was what was wrong with my loop, apparently allowing creation on `open()` also prevents `write()` from appending. when I changed it to `w` like yours it worked…