Preallocating Array in a Memory Efficient Way

Hi,

I need to work with a very big Array

z=zeros(Int64,2^K,2^K,K,T)

What is the proper way of pre-allocating this Array in such a way that it uses the least memory possible when K is large?

How big is K? Will most of the elements be 0? If so, using a sparse data structure will be helpful. Also, are you sure you need to store the array? If you can get the same effect from looping over a small portion of it, that should be much more efficient.

2 Likes

K is around 15! Yes, most elements will be 0. However, most of this are 3-dimensional arrays and I could not get sparse structures to work in this case. A MWE of my problem can be found here Performance Tips for Combinatorial Problem

Is there any structure to it? Blocks, for instance?

Yes, you can think of z=zeros(Int64,2^K,2^K,K,T) as 2^K2^K matrices of size KT

If you don’t need all of the matrices at the same time, store them as an array of arrays, with possibly “nothing” entries.

1 Like