Hi,
I need to work with a very big Array
z=zeros(Int64,2^K,2^K,K,T)
What is the proper way of pre-allocating this Array in such a way that it uses the least memory possible when K is large?
How big is K? Will most of the elements be 0? If so, using a sparse data structure will be helpful. Also, are you sure you need to store the array? If you can get the same effect from looping over a small portion of it, that should be much more efficient.
2 Likes
K is around 15! Yes, most elements will be 0. However, most of this are 3-dimensional arrays and I could not get sparse structures to work in this case. A MWE of my problem can be found here Performance Tips for Combinatorial Problem
Is there any structure to it? Blocks, for instance?
Yes, you can think of z=zeros(Int64,2^K,2^K,K,T) as 2^K2^K matrices of size KT
If you don’t need all of the matrices at the same time, store them as an array of arrays, with possibly “nothing” entries.
1 Like