Excessive memory usage when adding millions of variables to a JuMP Model

Walter
Tue, 24 Jun 2025 13:02:48 +0000
Optimization (Mathematical)

Hi,

I’m solving a large MIP using JuMP + Gurobi. My approach dynamically adds variables to the model (similar to column generation). In some cases, up to 10 000 000 variables are added within a single step. During this step, the memory usage explodes. Julia consumes all available RAM on my desktop, and on a cluster with 16 GB per process I obtain a memory error. However, after this step, the model uses only around 1.2 GB according to Base.summarysize.
The model has roughly 1000 constraints and 50 million nonzeros. Profiling suggest that most of the memory is allocated within set_normalized_coefficient.

Why does adding variables cause such a high memory usage? Is this due to internal JuMP structures? Would building the model directly using the Gurobi C API (or MOI) help reduce memory consumption?
Here is a simplified example of my code:

using Random
using StatsBase
using JuMP
using Gurobi

function obtain_new_columns(num_nodes, num_routes)
    new_columns= Vector{Vector{Int64}}()

    for i=1:num_routes
        route= sample(1:num_nodes, 20; replace= false)
        push!(new_columns, route)
    end

    return new_columns
end

Random.seed!(1234)
num_nodes= 100
model = direct_model(Gurobi.Optimizer())
set_string_names_on_creation(model, false)

x= Vector{VariableRef}()
@constraint(model, constr[i=1:num_nodes], 0 == 1)
for num_routes in [10000, 9990000]#while true
    new_routes= obtain_new_columns(num_nodes, num_routes)
    for route in new_routes
        push!(x, @variable(model, binary=true))

        set_objective_coefficient(model, x[end], 1)
        for i in route
            set_normalized_coefficient(constr[i], x[end], 1)
        end
    end
end
println("Memory usage model:", round(Base.summarysize(model)/1024^3, digits=2), "GB")

Thanks a lot!

13 posts - 5 participants