# Very large arrays

**URL:** <https://discourse.julialang.org/t/very-large-arrays/77942>\
**Category:** General Usage\
**Created:** [March 16, 2022, 4:24am UTC](https://discourse.julialang.org/t/very-large-arrays/77942 "2022-03-16T04:24:46Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lilith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lilith/32/27492_2.png) [@Lilith](https://discourse.julialang.org/u/Lilith)\
**Post date:** [March 16, 2022, 4:24am UTC](https://discourse.julialang.org/t/very-large-arrays/77942/1 "2022-03-16T04:24:46Z")

</div>

Does Julia support `AbstractArrays` larger than `typemax(Int)`? (e.g. sparse arrays or arrays larger than 2^31 on a 32-bit Julia install)

Is code in Base allowed to assume that an AbstractArray has length no more than `typemax(Int)`?

---

<div class="post-metadata">

**Author:** ![lawless-m](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lawless-m/32/30869_2.png) [@lawless-m](https://discourse.julialang.org/u/lawless-m)\
**Post date:** [March 16, 2022, 8:55am UTC](https://discourse.julialang.org/t/very-large-arrays/77942/2 "2022-03-16T08:55:21Z")

</div>

```julia
julia> v = Vector{Char}(undef, typemax(Int))
ERROR: invalid Array size

julia> v = Vector{Char}(undef, typemax(UInt32))
4294967295-element Vector{Char}:
...

julia> v = Vector{Char}(undef, 16typemax(UInt32))
ERROR: OutOfMemoryError() # I have 64Gb

```

ERROR: invalid Array size

is generated by

[https://github.com/JuliaLang/julia/blob/master/src/array.c#L107](https://github.com/JuliaLang/julia/blob/master/src/array.c#L107)

while the actual check `jl_array_validate_dims` is done in

[https://github.com/JuliaLang/julia/blob/master/src/array.c#L79-L96](https://github.com/JuliaLang/julia/blob/master/src/array.c#L79-L96)

and that error is from

```julia
   wideint_t prod = (wideint_t)elsz * (wideint_t)_nel;
   if (prod >= (wideint_t) MAXINTVAL)
        return 2;

```

---

<div class="post-metadata">

**Author:** ![Per](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/per/32/10387_2.png) [@Per](https://discourse.julialang.org/u/Per)\
**Post date:** [March 16, 2022, 9:18am UTC](https://discourse.julialang.org/t/very-large-arrays/77942/3 "2022-03-16T09:18:10Z")

</div>

An `AbstractArray` can be arbitrarily large. `LinearAlgebra.I`, for example, is infinite in the sense that we can access `I[i,j]` for any positive integers `I` and `j`.

```julia
using LinearAlgebra
I[big(2)^100,big(2)^100] # returns true

```

Hence, one cannot in general assume that every `AbstractArray` has a `length`.

```julia
length(I) # throws a MethodError

```

Edit: Hmm. Actually:

```julia
I isa AbstractArray # returns false

```

Forget I said anything.

---

<div class="post-metadata">

**Author:** ![Lilith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lilith/32/27492_2.png) [@Lilith](https://discourse.julialang.org/u/Lilith)\
**Post date:** [March 16, 2022, 2:15pm UTC](https://discourse.julialang.org/t/very-large-arrays/77942/4 "2022-03-16T14:15:15Z")

</div>

I know `Vector`s must be smaller than `typemax(Int)`, but what about `AbstractArray`s?

---

<div class="post-metadata">

**Author:** ![rfourquet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rfourquet/32/3610_2.png) [@rfourquet](https://discourse.julialang.org/u/rfourquet)\
**Post date:** [March 16, 2022, 4:02pm UTC](https://discourse.julialang.org/t/very-large-arrays/77942/5 "2022-03-16T16:02:49Z")

</div>

`UnitRange` is an example of array which can be large, e.g. `1:typemax(Int128)`.

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [March 16, 2022, 5:18pm UTC](https://discourse.julialang.org/t/very-large-arrays/77942/6 "2022-03-16T17:18:51Z")

</div>

According to the [abstract array interface](https://docs.julialang.org/en/v1/manual/interfaces/#man-interface-array) arrays only need to support indexing with `Int`.

The docs for `Base.to_index` say:

> ```julia
> to_index(i)
> 
> ```
> 
> Convert index `i` to an `Int` or array of `Int`s to be used as an index for all arrays.
> 
> Custom index types may specialize `to_index(::CustomIndex)` to provide special indexing behaviors. This must return either an `Int` or an `AbstractArray` of `Int`s.

And indeed `[][typemax(UInt128)]` throws an error from `to_index`.