How to pass a random array of 2^57 as an input to a function?

General Usage
1

I am creating a function that needs a very large size of the array as an input. Please let me know how can I do that.

2

Why would it be different from passing a smaller random array?

2 Likes
3

wait, how much RAM you have? an array of length 2^57 is like…:

julia> 2^57 / 1024^5
128

128EB Pb at least, assuming it’s a bitarray

6 Likes
4

It is a part of parallel computing, and machines are huge

5

No difference, I am ok if anyone has an idea to pass an array like 2^3. And later for load testing I can change to 2^57

6

This reminds me of an earlier discussion. What do you mean with “random array”? In case you mean an array of random numbers: there is no need to pass that array, instead you can pass the random seed and generate the numbers anew inside the function.

1 Like
7

Yes, I need to use Seed also, but I am using recursion in the function, so I have to use array as a parameter.

I am working on creating a sum reduction on large array with limited number of tasks , lets say 20, in parallel.

8

I don’t think the world anyone has 128EB 128Pb of RAM.

You just cannot have a 2^57 array “concretely exist” at any given moment

4 Likes
9

This is fine. Passing an Array as an argument to a function does not copy the Array in memory. No special treatment is required.

1 Like
10

You’re off on your prefixes. That’s 128 PiB.

For comparison the world top supercomputer has 5 PiB of memory. Granted that it’s top ranked on flops and not memory I still think it’s safe to assume that nobody has access to any kind of hardware that can handle an array with 2^57 elements.

5 Likes
12

Presumably the function does not need all of it in memory at once? So you can break it up into pieces.

13

If it is a random array, I would pass a seed of the random number generator and calculate elements only when I need them.

2 Likes
14

Thanks

15

It is a part of sum reduction and every call will be run parallelly.