Sometimes cannot reclaim memory by manual GC.gc()

General Usage
,
1

Edit: Please see the #7 post directly if you’re interested. Thanks.

The idea is: I at first create a Vector

v = Vector{Matrix{Float64}}(undef, 100);

Then I need to fill it with some concrete objects (here are matrices):

for i = 1:100
   v[i] = rand(9999,9999)
end

These operations will use much (say 60GB) memory.
Suppose now I’ve finished using the entry matrices and now I want to reclaim my memory.
I can do

dumb = rand(3, 3)
for i = 1:100
    v[i] = dumb
end
GC.gc()

Then the memory occupation will drop to the initial level (say, 60GB are freed).

The question now is that the above procedure appears not to be effective for JuMP’s direct_models (with the common GRB_ENV).

const GRB_ENV = Gurobi.Env();
const m = Vector{JuMP.Model}(undef, S); # S is some large number (of scenarios)
# fill `m` with `S` concrete large distinct models, which are all initialized by this function
function model(GRB_ENV)
    m = JuMP.direct_model(Gurobi.Optimizer(GRB_ENV))
    JuMP.set_attribute(m, "OutputFlag", 0)
    JuMP.set_attribute(m, "Threads", 1)
    JuMP.set_attribute(m, "Method", 2)
    JuMP.set_attribute(m, "Crossover", 0)
    m
end;
# then create a common dumb model also using the above function
# then flush the `m` with the dumb model
# I fail to see my memory reclaimed, why?
2

According to my observation from the htop in zsh, the occupation at three instances are illustrated below

const S = 256
const GRB_ENV = Gurobi.Env();
const m = Vector{JuMP.Model}(undef, S);
function my_initialize(GRB_ENV)
    m = JuMP.direct_model(Gurobi.Optimizer(GRB_ENV))
    JuMP.set_attribute(m, "OutputFlag", 0)
    JuMP.set_attribute(m, "Threads", 1)
    JuMP.set_attribute(m, "Method", 2)
    JuMP.set_attribute(m, "Crossover", 0)
    m
end
function fill_with_concrete!(m, s, Data)
    model = my_initialize(GRB_ENV)
    x = add_decision_and_constrs!(model, s, Data) # this operation will allocate much memory
    m[s] = model
end

# Mem[873M/256G]

foreach(wait, [Threads.@spawn(fill_with_concrete!(m, s, Data)) for s=1:S])

# Mem[4.68G/256G]

dumb = my_initialize(GRB_ENV)
for s=1:S
    m[s] = dumb
end
GC.gc()

# Mem[4.62G/256G]
3

The Gurobi.Env object is not thread-safe.

I guess I need to update:

In addition, you must not simultaneously create multiple models using the same environment.

Edit: Update thread-safety notes for Gurobi.Env by odow · Pull Request #665 · jump-dev/Gurobi.jl · GitHub

Otherwise: this code should work. If memory isn’t being freed, you must have a reference to the model somewhere that is preventing the GC from freeing it. You can see how many models are attached to the environment using GRB_ENV.attached_models.

4

The behavior is now 858M → 4.69G → 4.65G, even if I’ve removed the GRB_ENV dependency (see the cyan arrow below).

image
image1131×965 50.7 KB

5

I even further removed the Gurobi dependency and work with JuMP.Model() now.

The major memory is still unreclaimed (793M → 3.14G → 2.73G)

image
image1137×994 52.7 KB

If I further enlarge S from 256 to 2560, then memory pattern now is 878M → 11.2G → 9.21G

6

Try running GC.gc() multiple times. And add a return nothing to the end of perfect_consts!. You must have a reference somewhere that is preventing the models from being garbage collected.

Does it happen if you remove Threads.@spawn?

I don’t think this is related to JuMP.

7

I agree.

I find that the manual GC.gc() appears to be nondeterministic, via this self-contained general example:

function f!(v)
    m = v[end]
    for i = eachindex(v)
        v[i] = m
    end
end;
# 662M here
v = map(fetch, [Threads.@spawn(rand(333,333)) for _ = 1:10000]);
# 8.91G here
f!(v)
GC.gc()
# 8.91G here
v1 = map(fetch, [Threads.@spawn(rand(333,555)) for _ = 1:10000]);
# 22.1G here
f!(v1)
GC.gc()
# 999M here

The 5 comments are observed from my htop in zsh.