OutOfMemoryError() on large multicommodity flow model

Specific Domains Optimization (Mathematical)
1

I ran the code below:

model = Model(HiGHS.Optimizer)

# Generate a flow variable for each item in df_flowslist1 which is ~500k items
df_flowslist1.x_flow = @variable(model, x[1:size(df_flowslist1, 1)] >= 0)

#generate a supply variable for each row in supplylist
supplydata.y_supply = @variable(model, y[1:size(supplydata, 1)] >= 0)

# Note this constrains output below capacity effectively Mass Balance 1 from the last model
set_upper_bound.(supplydata.y_supply, supplydata.prod_capa_Mt)
#in original model edgelist is all the edges not broken out by coal types
# note these statements will not accept zero values so have to fix them using coalesce
# other issue is how to differentiate between electric and coal physical flows

@objective(model, Min,
    sum(df_flowslist1.x_flow' .* df_flowslist1.transp_cost_tot_usd) +
    sum(supplydata.y_supply' .* supplydata.total_cash_cost_ex_transp) +
    sum(df_flowslist1.x_flow' .* df_flowslist1.CV_PJ_p_Mt_therm .* df_flowslist1.conversion_eff') * 10^6
    )

And hit the following error:

OutOfMemoryError()

Stacktrace:
  [1] Array
    @ .\boot.jl:479 [inlined]
  [2] Array
    @ .\boot.jl:487 [inlined]
  [3] Array
    @ .\boot.jl:494 [inlined]
  [4] similar
    @ .\abstractarray.jl:884 [inlined]
  [5] similar
    @ .\abstractarray.jl:883 [inlined]
  [6] similar
    @ .\broadcast.jl:212 [inlined]
  [7] similar
    @ .\broadcast.jl:211 [inlined]
  [8] copy
    @ .\broadcast.jl:898 [inlined]
  [9] materialize(bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{2}, Nothing, typeof(*), Tuple{LinearAlgebra.Adjoint{VariableRef, Vector{VariableRef}}, Vector{Float64}}})
    @ Base.Broadcast .\broadcast.jl:873
 [10] macro expansion
    @ C:\Users\aturn\.julia\packages\MutableArithmetics\K9YPJ\src\rewrite.jl:321 [inlined]
 [11] macro expansion
    @ C:\Users\aturn\.julia\packages\JuMP\D44Aq\src\macros.jl:717 [inlined]
 [12] top-level scope
    @ In[26]:15

Any ideas on what to do here? df_flowslist is pretty big - about 150,000 rows because it is a multicommodity flow problem with several different products.

If i remove the " ’ " and specify the problem as below:

@objective(model, Min,
    sum(df_flowslist1.x_flow .* df_flowslist1.transp_cost_tot_usd) +
    sum(supplydata.y_supply .* supplydata.total_cash_cost_ex_transp) +
    sum(df_flowslist1.x_flow .* df_flowslist1.CV_PJ_p_Mt_therm .* df_flowslist1.conversion_eff) * 10^6
    )

Then I get this error:

Error: Maximum call stack size exceeded


Debugging tips: use 'unpacked/MathJax.js', inspect 'MathJax.Hub.lastError' in the browser console
2

For the first part:

Instead of

sum(df_flowslist1.x_flow' .* df_flowslist1.transp_cost_tot_usd)

do

sum(df_flowslist1.x_flow .* df_flowslist1.transp_cost_tot_usd)

or

df_flowslist1.x_flow' * df_flowslist1.transp_cost_tot_usd

The first computes the pairwise product between the two vectors and then sums them up. The second computes the inner product between the two vectors. Both are equivalent.

Your original code tries to transpose the first vector, then compute the pairwise product and then sum. I don’t understand the error, but I guess Julia uses recursion somewhere, or it falls back to some very inefficient algorithm.

For the second part:

This error looks like it comes from MathJax, not from JuMP or Julia. Try ending the cell with a ; so that printing is disabled:

@objective(model, Min,
    sum(df_flowslist1.x_flow .* df_flowslist1.transp_cost_tot_usd) +
    sum(supplydata.y_supply .* supplydata.total_cash_cost_ex_transp) +
    sum(df_flowslist1.x_flow .* df_flowslist1.CV_PJ_p_Mt_therm .* df_flowslist1.conversion_eff) * 10^6
    );
3

Thanks very much seems the semicolon was key here

image
image814×402 27 KB

4

I’ve opened an issue: Truncate printing expressions if they are very large · Issue #3574 · jump-dev/JuMP.jl · GitHub.

We should probably truncate the printing. I’m sure that you’re not the first person that has hit this.

2 Likes