Very large arrays

General Usage
1

Does Julia support AbstractArrays larger than typemax(Int)? (e.g. sparse arrays or arrays larger than 2^31 on a 32-bit Julia install)

Is code in Base allowed to assume that an AbstractArray has length no more than typemax(Int)?

2 
julia> v = Vector{Char}(undef, typemax(Int))
ERROR: invalid Array size

julia> v = Vector{Char}(undef, typemax(UInt32))
4294967295-element Vector{Char}:
...

julia> v = Vector{Char}(undef, 16typemax(UInt32))
ERROR: OutOfMemoryError() # I have 64Gb

ERROR: invalid Array size

is generated by

https://github.com/JuliaLang/julia/blob/master/src/array.c#L107

while the actual check jl_array_validate_dims is done in

https://github.com/JuliaLang/julia/blob/master/src/array.c#L79-L96

and that error is from

   wideint_t prod = (wideint_t)elsz * (wideint_t)_nel;
   if (prod >= (wideint_t) MAXINTVAL)
        return 2;
3

An AbstractArray can be arbitrarily large. LinearAlgebra.I, for example, is infinite in the sense that we can access I[i,j] for any positive integers I and j.

using LinearAlgebra
I[big(2)^100,big(2)^100] # returns true

Hence, one cannot in general assume that every AbstractArray has a length.

length(I) # throws a MethodError

Edit: Hmm. Actually:

I isa AbstractArray # returns false

Forget I said anything.

4

I know Vectors must be smaller than typemax(Int), but what about AbstractArrays?

5

UnitRange is an example of array which can be large, e.g. 1:typemax(Int128).

1 Like
6

According to the abstract array interface arrays only need to support indexing with Int.

The docs for Base.to_index say:

to_index(i)

Convert index i to an Int or array of Ints to be used as an index for all arrays.

Custom index types may specialize to_index(::CustomIndex) to provide special indexing behaviors. This must return either an Int or an AbstractArray of Ints.

And indeed [][typemax(UInt128)] throws an error from to_index.