# Where is the excessive memory consumption

**URL:** <https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174>\
**Category:** Performance\
**Tags:** jump\
**Created:** [September 7, 2024, 5:54pm UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174 "2024-09-07T17:54:26Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Betristor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/betristor/32/45264_2.png) [@Betristor](https://discourse.julialang.org/u/Betristor)\
**Post date:** [September 7, 2024, 5:54pm UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/1 "2024-09-07T17:54:26Z")

</div>

I have a JuMP model with about 0.2 billion variables. Gurobi reports 50 GB of memory consumption and htop reports consistent 350 GB of memory consumption during solving.

I’m most curious about the excessive memory consumption between htop (total process) and gurobi (model itself). Why is there a 7x difference?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [September 7, 2024, 5:55pm UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/2 "2024-09-07T17:55:57Z")

</div>

are you looking at virtual or resident memory? virtual memory is mostly fake.

---

<div class="post-metadata">

**Author:** ![Betristor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/betristor/32/45264_2.png) [@Betristor](https://discourse.julialang.org/u/Betristor)\
**Post date:** [September 7, 2024, 6:37pm UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/3 "2024-09-07T18:37:09Z")

</div>

I looked up the Resident Memory Size part.

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [September 8, 2024, 10:54pm UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/4 "2024-09-08T22:54:54Z")

</div>

Are you using `direct_model`?

If you are using `Model`, there can be multiple copies of the JuMP model in memory.

Note that 200 million variables is at the upper limit of what solvers can solve. This is an exceedingly large problem.

---

<div class="post-metadata">

**Author:** ![Betristor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/betristor/32/45264_2.png) [@Betristor](https://discourse.julialang.org/u/Betristor)\
**Post date:** [September 9, 2024, 6:01am UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/5 "2024-09-09T06:01:57Z")

</div>

Yes, I changed the `Model` constructor to `direct_model`. So I’m wondering why is there an excessive memory consumption.

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [September 9, 2024, 6:31am UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/6 "2024-09-09T06:31:06Z")

</div>

Its impossible to say without seeing your code.

Did you turn off string names?

---

<div class="post-metadata">

**Author:** ![Betristor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/betristor/32/45264_2.png) [@Betristor](https://discourse.julialang.org/u/Betristor)\
**Post date:** [September 9, 2024, 8:00am UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/7 "2024-09-09T08:00:07Z")

</div>

No, I did not manipulate string names.

---

<div class="post-metadata">

**Author:** ![Betristor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/betristor/32/45264_2.png) [@Betristor](https://discourse.julialang.org/u/Betristor)\
**Post date:** [September 10, 2024, 2:37am UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/8 "2024-09-10T02:37:21Z")

</div>

In a toy example, the model memory consumtion is summarized as follows:

| Report Source | Consumption | Description |
| --- | --- | --- |
| gurobi | 12 MB | |
| summarysize | 34.5 MB | with string names |
| summarysize | 31.1 MB | without string names |

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [September 10, 2024, 3:18am UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/9 "2024-09-10T03:18:20Z")

</div>

> Did you turn off string names?

I meant using

> **[Performance tips · JuMP](https://jump.dev/JuMP.jl/stable/tutorials/getting_started/performance_tips/#Disable-string-names)**
>
> Documentation for JuMP.

I’m not sure if there is anything actionable to do here. You’re solving a large MIP. It takes a lot of memory. I can’t see the code that you are running.

> Gurobi reports 50 GB of memory consumption

Where did Gurobi report this?

---

<div class="post-metadata">

**Author:** ![Betristor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/betristor/32/45264_2.png) [@Betristor](https://discourse.julialang.org/u/Betristor)\
**Post date:** [September 10, 2024, 3:22am UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/10 "2024-09-10T03:22:22Z")

</div>

Yeah, I used `set_string_names_on_creation‎` to control the string names. Gurobi will report model statistics using barrier algorithm (Method=2).

```julia
Barrier statistics:
 Dense cols : 7797
 Free vars : 289080
 AA' NZ : 1.198e+08
 Factor NZ : 5.842e+08 (roughly 45.0 GB of memory)
 Factor Ops : 2.087e+12 (roughly 6 seconds per iteration)
 Threads : 32

```

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [September 10, 2024, 3:29am UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/11 "2024-09-10T03:29:05Z")

</div>

That’s only how much memory it spent factoring the matrix. That’s not the total amount of memory that it consumes during the solve.

---

<div class="post-metadata">

**Author:** ![Betristor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/betristor/32/45264_2.png) [@Betristor](https://discourse.julialang.org/u/Betristor)\
**Post date:** [September 10, 2024, 3:53am UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/12 "2024-09-10T03:53:36Z")

</div>

Well, that means I can’t figure out how much memory consumption the model takes. BTW, turning off the string names do help reduce some memory consumption.

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [September 10, 2024, 3:57am UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/13 "2024-09-10T03:57:34Z")

</div>

> [@Betristor](#):
>
> Well, that means I can’t figure out how much memory consumption the model takes

> htop reports consistent 350 GB of memory consumption during solving

Is there any reason to distrust this?

> BTW, turning off the string names do help reduce some memory consumption

Correct.

---

<div class="post-metadata">

**Author:** ![Betristor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/betristor/32/45264_2.png) [@Betristor](https://discourse.julialang.org/u/Betristor)\
**Post date:** [September 10, 2024, 6:20am UTC](https://discourse.julialang.org/t/where-is-the-excessive-memory-consumption/119174/14 "2024-09-10T06:20:53Z")

</div>

I thought the number htop reports is a measure for the total process including data loading, model construction and solving. But since htop reports dynamic memory consumption, I guess the number during solving represents the memory consumtion of the model.