# Very large arrays

Does Julia support `AbstractArrays` larger than `typemax(Int)`? (e.g. sparse arrays or arrays larger than 2^31 on a 32-bit Julia install)

Is code in Base allowed to assume that an AbstractArray has length no more than `typemax(Int)`?

``````julia> v = Vector{Char}(undef, typemax(Int))
ERROR: invalid Array size

julia> v = Vector{Char}(undef, typemax(UInt32))
4294967295-element Vector{Char}:
...

julia> v = Vector{Char}(undef, 16typemax(UInt32))
ERROR: OutOfMemoryError() # I have 64Gb
``````

ERROR: invalid Array size

is generated by

while the actual check `jl_array_validate_dims` is done in

and that error is from

``````   wideint_t prod = (wideint_t)elsz * (wideint_t)_nel;
if (prod >= (wideint_t) MAXINTVAL)
return 2;
``````

An `AbstractArray` can be arbitrarily large. `LinearAlgebra.I`, for example, is infinite in the sense that we can access `I[i,j]` for any positive integers `I` and `j`.

``````using LinearAlgebra
I[big(2)^100,big(2)^100] # returns true
``````

Hence, one cannot in general assume that every `AbstractArray` has a `length`.

``````length(I) # throws a MethodError
``````

Edit: Hmm. Actually:

``````I isa AbstractArray # returns false
``````

Forget I said anything.

I know `Vector`s must be smaller than `typemax(Int)`, but what about `AbstractArray`s?

`UnitRange` is an example of array which can be large, e.g. `1:typemax(Int128)`.

1 Like

According to the abstract array interface arrays only need to support indexing with `Int`.

The docs for `Base.to_index` say:

``````to_index(i)
``````

Convert index `i` to an `Int` or array of `Int`s to be used as an index for all arrays.

Custom index types may specialize `to_index(::CustomIndex)` to provide special indexing behaviors. This must return either an `Int` or an `AbstractArray` of `Int`s.

And indeed `[][typemax(UInt128)]` throws an error from `to_index`.