I would like to do a piece-wise reduction (on sub-arrays) of a large array that is too big to fit into memory. Is there a ‘storage-based array’ that implements @views?

The Mmap package provides memory-mapped file-based arrays that sound like they should do what you want. Only the portion you are accessing is paged into memory. (Every AbstractArray subtype supports @views and subarrays.)

Storage-based array that implements @views?

General Usage

mkarikom December 31, 2020, 10:43pm 8

Thanks!
Is the do construct required here? I used almost the same for-loop strategy in this post but if fails.

Topic		Replies	Views
Writing an array too large to store in memory Performance question , jld	2	2187	March 16, 2018
Use of Memory-mapped I/O General Usage memory , memory-allocation	9	2805	September 5, 2019
Growing mmaped arrays General Usage	6	627	December 17, 2018
Problem placing views into an array General Usage question , array , views	3	534	April 29, 2021
Memory allocation while using @views Performance performance , arrays	5	1209	August 16, 2021

Storage-based array that implements @views?

Related topics