Shrinking array size

General Usage
1

I have some vectors which are initially very big to do some computation, but which shrink afterwards and will stay small for the rest of the code’s execution. Unfortunately, it seems that Julia’s underlying arrays never really shrink:

julia> x = fill(1,100);

julia> ccall(:jl_array_size, Int, (Any, UInt), x, 1)
100

julia> empty!(x)
0-element Array{Int64,1}

julia> sizehint!(x, 0)
0-element Array{Int64,1}

julia> ccall(:jl_array_size, Int, (Any, UInt), x, 1)
100

This code from array.c seems to suggest that shrinking is possible:

JL_DLLEXPORT void jl_array_sizehint(jl_array_t *a, size_t sz)
{
    size_t n = jl_array_nrows(a);

    int min = a->offset + a->length;
    sz = (sz < min) ? min : sz;

    if (sz <= a->maxsize) {
        size_t dec = a->maxsize - sz;
        //if we don't save at least an eighth of maxsize then its not worth it to shrink
        if (dec < a->maxsize / 8) return;
        jl_array_shrink(a, dec);
    }
    else {
        size_t inc = sz - n;
        jl_array_grow_end(a, inc);

        a->nrows = n;
#ifdef STORE_ARRAY_LEN
        a->length = n;
#endif
    }
}

Unfortunately jl_array_shrink is unexported. I’ve been otherwise unable to see any reduction in jl_array_size. Am I misunderstanding something?

2

Here is the PR that implemented shrinking with sizehint! https://github.com/JuliaLang/julia/pull/26201. But it doesn’t seem to have tests for the capacity shrinking.

1 Like
3

Interesting, it seems that there’s a threshold around maxsize that I don’t understand:

julia> A = zeros(400_000);

julia> empty!(A)
0-element Array{Float64,1}

julia> sizehint!(A, 0)
0-element Array{Float64,1}

julia> ccall(:jl_array_size, Int, (Any, UInt), A, 1)
0   # worked

julia> A = zeros(400);

julia> empty!(A)
0-element Array{Float64,1}

julia> sizehint!(A, 0)
0-element Array{Float64,1}

julia> ccall(:jl_array_size, Int, (Any, UInt), A, 1)
400   # failed
4

Inline allocated buffer cannnot be shrinked.

1 Like
5

That makes sense. How big is that buffer?

6

It’s not a fixed size but an array with that buffer will never have that freed. It’s not something you need to worry about if you have some very big array initially.

2 Likes