I have a JuMP model with about 0.2 billion variables. Gurobi reports 50 GB of memory consumption and htop reports consistent 350 GB of memory consumption during solving.
I’m most curious about the excessive memory consumption between htop (total process) and gurobi (model itself). Why is there a 7x difference?
are you looking at virtual or resident memory? virtual memory is mostly fake.
I looked up the Resident Memory Size part.
Are you using direct_model?
If you are using Model, there can be multiple copies of the JuMP model in memory.
Note that 200 million variables is at the upper limit of what solvers can solve. This is an exceedingly large problem.
Yes, I changed the Model constructor to direct_model. So I’m wondering why is there an excessive memory consumption.
Its impossible to say without seeing your code.
Did you turn off string names?
No, I did not manipulate string names.
In a toy example, the model memory consumtion is summarized as follows:
| Report Source | Consumption | Description |
|---|---|---|
| gurobi | 12 MB | |
| summarysize | 34.5 MB | with string names |
| summarysize | 31.1 MB | without string names |
Did you turn off string names?
I meant using
I’m not sure if there is anything actionable to do here. You’re solving a large MIP. It takes a lot of memory. I can’t see the code that you are running.
Gurobi reports 50 GB of memory consumption
Where did Gurobi report this?
Yeah, I used set_string_names_on_creation to control the string names. Gurobi will report model statistics using barrier algorithm (Method=2).
Barrier statistics:
Dense cols : 7797
Free vars : 289080
AA' NZ : 1.198e+08
Factor NZ : 5.842e+08 (roughly 45.0 GB of memory)
Factor Ops : 2.087e+12 (roughly 6 seconds per iteration)
Threads : 32
That’s only how much memory it spent factoring the matrix. That’s not the total amount of memory that it consumes during the solve.
Well, that means I can’t figure out how much memory consumption the model takes. BTW, turning off the string names do help reduce some memory consumption.
htop reports consistent 350 GB of memory consumption during solving
Is there any reason to distrust this?
BTW, turning off the string names do help reduce some memory consumption
Correct.
I thought the number htop reports is a measure for the total process including data loading, model construction and solving. But since htop reports dynamic memory consumption, I guess the number during solving represents the memory consumtion of the model.