Cannot reclaim memory after building Gurobi direct models

Specific Domains Optimization (Mathematical)
,
1

The idea is: I at first create a Vector

v = Vector{Matrix{Float64}}(undef, 100);

Then I need to fill it with some concrete objects (here are matrices):

for i = 1:100
   v[i] = rand(9999,9999)
end

These operations will use much (say 60GB) memory.
Suppose now I’ve finished using the entry matrices and now I want to reclaim my memory.
I can do

dumb = rand(3, 3)
for i = 1:100
    v[i] = dumb
end
GC.gc()

Then the memory occupation will drop to the initial level (say, 60GB are freed).

The question now is that the above procedure appears not to be effective for JuMP’s direct_models (with the common GRB_ENV).

const GRB_ENV = Gurobi.Env();
const m = Vector{JuMP.Model}(undef, S); # S is some large number (of scenarios)
# fill `m` with `S` concrete large distinct models, which are all initialized by this function
function model(GRB_ENV)
    m = JuMP.direct_model(Gurobi.Optimizer(GRB_ENV))
    JuMP.set_attribute(m, "OutputFlag", 0)
    JuMP.set_attribute(m, "Threads", 1)
    JuMP.set_attribute(m, "Method", 2)
    JuMP.set_attribute(m, "Crossover", 0)
    m
end;
# then create a common dumb model also using the above function
# then flush the `m` with the dumb model
# I fail to see my memory reclaimed, why?
2

According to my observation from the htop in zsh, the occupation at three instances are illustrated below

const S = 256
const GRB_ENV = Gurobi.Env();
const m = Vector{JuMP.Model}(undef, S);
function my_initialize(GRB_ENV)
    m = JuMP.direct_model(Gurobi.Optimizer(GRB_ENV))
    JuMP.set_attribute(m, "OutputFlag", 0)
    JuMP.set_attribute(m, "Threads", 1)
    JuMP.set_attribute(m, "Method", 2)
    JuMP.set_attribute(m, "Crossover", 0)
    m
end
function fill_with_concrete!(m, s, Data)
    model = my_initialize(GRB_ENV)
    x = add_decision_and_constrs!(model, s, Data) # this operation will allocate much memory
    m[s] = model
end

# Mem[873M/256G]

foreach(wait, [Threads.@spawn(fill_with_concrete!(m, s, Data)) for s=1:S])

# Mem[4.68G/256G]

dumb = my_initialize(GRB_ENV)
for s=1:S
    m[s] = dumb
end
GC.gc()

# Mem[4.62G/256G]
3

The Gurobi.Env object is not thread-safe.

I guess I need to update:

In addition, you must not simultaneously create multiple models using the same environment.

Edit: Update thread-safety notes for Gurobi.Env by odow · Pull Request #665 · jump-dev/Gurobi.jl · GitHub

Otherwise: this code should work. If memory isn’t being freed, you must have a reference to the model somewhere that is preventing the GC from freeing it. You can see how many models are attached to the environment using GRB_ENV.attached_models.