# OutOfMemoryError() on large multicommodity flow model

**URL:** <https://discourse.julialang.org/t/outofmemoryerror-on-large-multicommodity-flow-model/106286>\
**Category:** Optimization (Mathematical)\
**Tags:** jump\
**Created:** [November 16, 2023, 12:37am UTC](https://discourse.julialang.org/t/outofmemoryerror-on-large-multicommodity-flow-model/106286 "2023-11-16T00:37:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nemoincognito](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nemoincognito/32/52395_2.png) [@nemoincognito](https://discourse.julialang.org/u/nemoincognito)\
**Post date:** [November 16, 2023, 12:37am UTC](https://discourse.julialang.org/t/outofmemoryerror-on-large-multicommodity-flow-model/106286/1 "2023-11-16T00:37:13Z")

</div>

I ran the code below:

```julia
model = Model(HiGHS.Optimizer)

# Generate a flow variable for each item in df_flowslist1 which is ~500k items
df_flowslist1.x_flow = @variable(model, x[1:size(df_flowslist1, 1)] >= 0)

#generate a supply variable for each row in supplylist
supplydata.y_supply = @variable(model, y[1:size(supplydata, 1)] >= 0)

# Note this constrains output below capacity effectively Mass Balance 1 from the last model
set_upper_bound.(supplydata.y_supply, supplydata.prod_capa_Mt)
#in original model edgelist is all the edges not broken out by coal types
# note these statements will not accept zero values so have to fix them using coalesce
# other issue is how to differentiate between electric and coal physical flows

@objective(model, Min,
    sum(df_flowslist1.x_flow' .* df_flowslist1.transp_cost_tot_usd) +
    sum(supplydata.y_supply' .* supplydata.total_cash_cost_ex_transp) +
    sum(df_flowslist1.x_flow' .* df_flowslist1.CV_PJ_p_Mt_therm .* df_flowslist1.conversion_eff') * 10^6
    )

```

And hit the following error:

```julia
OutOfMemoryError()

Stacktrace:
  [1] Array
    @ .\boot.jl:479 [inlined]
  [2] Array
    @ .\boot.jl:487 [inlined]
  [3] Array
    @ .\boot.jl:494 [inlined]
  [4] similar
    @ .\abstractarray.jl:884 [inlined]
  [5] similar
    @ .\abstractarray.jl:883 [inlined]
  [6] similar
    @ .\broadcast.jl:212 [inlined]
  [7] similar
    @ .\broadcast.jl:211 [inlined]
  [8] copy
    @ .\broadcast.jl:898 [inlined]
  [9] materialize(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{2}, Nothing, typeof(*), Tuple{LinearAlgebra.Adjoint{VariableRef, Vector{VariableRef}}, Vector{Float64}}})
    @ Base.Broadcast .\broadcast.jl:873
 [10] macro expansion
    @ C:\Users\aturn\.julia\packages\MutableArithmetics\K9YPJ\src\rewrite.jl:321 [inlined]
 [11] macro expansion
    @ C:\Users\aturn\.julia\packages\JuMP\D44Aq\src\macros.jl:717 [inlined]
 [12] top-level scope
    @ In[26]:15

```

Any ideas on what to do here? df\_flowslist is pretty big - about 150,000 rows because it is a multicommodity flow problem with several different products.

If i remove the " ’ " and specify the problem as below:

```julia
@objective(model, Min,
    sum(df_flowslist1.x_flow .* df_flowslist1.transp_cost_tot_usd) +
    sum(supplydata.y_supply .* supplydata.total_cash_cost_ex_transp) +
    sum(df_flowslist1.x_flow .* df_flowslist1.CV_PJ_p_Mt_therm .* df_flowslist1.conversion_eff) * 10^6
    )

```

Then I get this error:

```julia
Error: Maximum call stack size exceeded

Debugging tips: use 'unpacked/MathJax.js', inspect 'MathJax.Hub.lastError' in the browser console

```

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [November 16, 2023, 12:57am UTC](https://discourse.julialang.org/t/outofmemoryerror-on-large-multicommodity-flow-model/106286/2 "2023-11-16T00:57:02Z")

</div>

For the first part:

Instead of

```julia
sum(df_flowslist1.x_flow' .* df_flowslist1.transp_cost_tot_usd)

```

do

```julia
sum(df_flowslist1.x_flow .* df_flowslist1.transp_cost_tot_usd)

```

or

```julia
df_flowslist1.x_flow' * df_flowslist1.transp_cost_tot_usd

```

The first computes the pairwise product between the two vectors and then sums them up. The second computes the inner product between the two vectors. Both are equivalent.

Your original code tries to transpose the first vector, then compute the pairwise product and then sum. I don’t understand the error, but I guess Julia uses recursion somewhere, or it falls back to some very inefficient algorithm.

For the second part:

This error looks like it comes from MathJax, not from JuMP or Julia. Try ending the cell with a `;` so that printing is disabled:

```julia
@objective(model, Min,
    sum(df_flowslist1.x_flow .* df_flowslist1.transp_cost_tot_usd) +
    sum(supplydata.y_supply .* supplydata.total_cash_cost_ex_transp) +
    sum(df_flowslist1.x_flow .* df_flowslist1.CV_PJ_p_Mt_therm .* df_flowslist1.conversion_eff) * 10^6
    );

```

---

<div class="post-metadata">

**Author:** ![nemoincognito](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nemoincognito/32/52395_2.png) [@nemoincognito](https://discourse.julialang.org/u/nemoincognito)\
**Post date:** [November 16, 2023, 1:17am UTC](https://discourse.julialang.org/t/outofmemoryerror-on-large-multicommodity-flow-model/106286/3 "2023-11-16T01:17:17Z")

</div>

Thanks very much seems the semicolon was key here

 ![image](https://global.discourse-cdn.com/julialang/original/3X/6/b/6bc05244feaed8159aa37d889b8594a2487c76d0.png)

---

<div class="post-metadata">

**Author:** ![odow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/odow/32/28685_2.png) [@odow](https://discourse.julialang.org/u/odow)\
**Post date:** [November 16, 2023, 1:30am UTC](https://discourse.julialang.org/t/outofmemoryerror-on-large-multicommodity-flow-model/106286/4 "2023-11-16T01:30:40Z")

</div>

I’ve opened an issue: [Truncate printing expressions if they are very large · Issue #3574 · jump-dev/JuMP.jl · GitHub](https://github.com/jump-dev/JuMP.jl/issues/3574).

We should probably truncate the printing. I’m sure that you’re not the first person that has hit this.