I want to reduce my model memory consumption with JuMP and Gurobi as solver. With about 0.3 billion variables in the model, the memory consumption increases to 1.5 TB, which exceeds my machine’s physical limit. JuMP supports to construct the model with a lower precision like Float32 but I don’t know whether Gurobi supports that since little documentation exists in this aspect. Could anyone provide some help?
No, it does not.
Note that even if you could build your problem, it is unlikely to solve in any reasonable amount of time. It is too large.
Thanks for your reply. Thanks♪(･ω･)ﾉ
Out of curiosity, where did you find a model so large?
I’m doing an optimization problem of energy system planning with over 25,000 renewable resources in spatial representation and a yearly horizon in temporal aspect, leading to over 1e8 variables.
Yeah if that’s an integer program you’re gonna have to reduce or decompose it somehow. I’m not even sure that linear programs this big can be solved at all
At present I keep the model linear by relaxing the unit commit constraints and intend to introduce integer variables gradually. As for the solving part, Gurobi could solve this problem in about 10 hours.
1 Like
Are you using direct_model?
@odow No, just the Model() constructor. Will direct_model() save some memory consumption?
1 Like
Yes, approximately half of the model size (not total, because solve uses some).
See Models · JuMP
2 Likes
w(ﾟДﾟ)w, that’s an incredible reduction especially for my situation. I see that warning says 1.the solution query may experience some erros if I modify the model and 2.some constraints may not be supported.
I’m not quite sure about the modify the model part. Does this mean solving the problem first and then modifying the model for another run?
For a linear model, I guess 2 will not affect my experience since linear programming is well supported by Gurobi or HiGHS.
You can ignore 1 if you are using Gurobi.
For 2, this is only a problem if you have interval constraints. But an error will be thrown if you do.
Well I do have a lot of box constraints but I constructed them seperately. The model construction process is successful. However, when I try to delete some constraints and unregister them from the model, the process was stuck using direct_model. Model constructor takes about 5 secs to do this while direct_model fails.
I managed to work out a minimum example.
using JuMP
using Dates
using Gurobi
Z = 3
G = 2100
T = 8760
m = Model(Gurobi.Optimizer)
println(Dates.now(), " Model")
@variable(m, 0 <= x[1:G, 1:T] <= 1)
println(Dates.now(), " Variable")
@expression(m, y[z in 1:Z, t in 1:T], sum(x[g, t] for g in filter(g -> g % z == 0, 1:G)))
@expression(m, obj, sum(rand() * m[:y][z, t] for z in 1:Z, t in 1:T))
println(Dates.now(), " Expression")
@constraint(m, c1, sum(m[:x][z, t] for z in 1:Z, t in 1:T) == 18000)
@constraint(m, c2[z in 1:Z], sum(m[:x][z, t] for t in 1:T) == 6000)
@constraint(m, c3[t in 1:T], sum(m[:x][z, t] for z in 1:Z) == 3)
println(Dates.now(), " Constraint")
@objective(m, Max, obj)
println(Dates.now(), " Objective")
delete.(m, m[:c3])
println(Dates.now(), " Deletion")
unregister(m, :c3)
The above example performs deletion successfully while direct_model did not. Test machine is a MacOS with 32GB RAM.
using JuMP
using Dates
using Gurobi
Z = 3
G = 2100
T = 8760
m = direct_model(Gurobi.Optimizer())
println(Dates.now(), " Model")
@variable(m, 0 <= x[1:G, 1:T] <= 1)
println(Dates.now(), " Variable")
@expression(m, y[z in 1:Z, t in 1:T], sum(x[g, t] for g in filter(g -> g % z == 0, 1:G)))
@expression(m, obj, sum(rand() * m[:y][z, t] for z in 1:Z, t in 1:T))
println(Dates.now(), " Expression")
@constraint(m, c1, sum(m[:x][z, t] for z in 1:Z, t in 1:T) == 18000)
@constraint(m, c2[z in 1:Z], sum(m[:x][z, t] for t in 1:T) == 6000)
@constraint(m, c3[t in 1:T], sum(m[:x][z, t] for z in 1:Z) == 3)
println(Dates.now(), " Constraint")
@objective(m, Max, obj)
println(Dates.now(), " Objective")
delete.(m, m[:c3])
println(Dates.now(), " Deletion")
unregister(m, :c3)
I cannot reproduce. What goes wrong? Is there an error?
No, it just stuck on my Mac. The direct_model version never made it to the deletion step.
The deletion is quick and successful using Model(). The logs are attached
2024-08-01T10:45:08.851 Model
2024-08-01T10:45:25.116 Variable
2024-08-01T10:45:41.674 Expression
2024-08-01T10:45:42.042 Constraint
2024-08-01T10:45:47.135 Objective
2024-08-01T10:45:47.282 Deletion
While direct_model version neither reports an error nor performs deletion step. I tested this on my Mac and a linux and direct_model just stuck.
Try the batched deletion:
delete(m, m[:c3])
Note the missing ..
Deleting a constraint is an expensive operation, and you are doing a lot of them.
As a side note: why do you need to delete constraints?
I’m curious about the difference between two versions. Model version performs deletion very fast.
I constructed a general model for energy system optimization and modify some constraints according to my specific scenarios. The motivations lay that basic modeling is written in a seperate Module and could be applied to many different cases with minor modifications.
By the way, I have experienced an error that using batch deletion on multi-indexed constraints crushed while element-wise operation worked long ago. Since that I tend to use dot operation with deletion.
ERROR: MethodError: no method matching delete(::Model, ::Matrix{ConstraintRef{Model, MathOptInterface.ConstraintIndex{…}, ScalarShape}})
Closest candidates are:
delete(::Model, ::NonlinearConstraintRef)
@ JuMP ~/.julia/packages/JuMP/7rBNn/src/nlp.jl:555
delete(::GenericModel, ::Vector{<:GenericVariableRef})
@ JuMP ~/.julia/packages/JuMP/7rBNn/src/variables.jl:513
delete(::GenericModel, ::Vector{<:ConstraintRef})
@ JuMP ~/.julia/packages/JuMP/7rBNn/src/constraints.jl:610
...