Best practice for large-scale models in JuMP?

Specific Domains Optimization (Mathematical)
,
1

Hi there,

We are translating a large scale LP model from GAMS to Julia/JuMP.
Large-scale means, there are a number of sets (regions, years, sub-annual time resolution - days and hours, technologies, etc.) and 2-5 dimensional variables, and the matrix (before aggregation by CPLEX) might be 5e+7 by 5e+7 with 3e+7 non-zeros. It is solvable in GAMS and Pyomo. Would be great to have JuMP version.

Here is an example of a small model which works well: UTOPIA_BASE_JuMP.7z

But we have encountered a limitation of this approach. It doesn’t work for large-scale models.

First, it seems to be not the best from performance perspective.

Second, there is some limitation in length of dictionaries (see my other question: #37247)

The sets in our model are character vectors (“Symbols” & “Dictionaries” in Julia). And the preference would be keep it this way, using names of regions etc. rather than numbered sets. Though performance is the priority. Would be sparse arrays a better alternative to Dicts/Tuples?

Any thoughts on how to define large-scale models and improve time-performance will be appreciated.

1 Like
2

I don’t think you will have any issue using “Symbols” & “Dictionaries” in Julia and in JuMP. I recomend starting with resolving #37247, this would be a Julia-only issue to resolve. Once that is working, then I would work on the scalability of the JuMP model.

I personally have never run into any scalability issues with JuMP. So I think it is worth the effort to work through any issues you might be experiencing.

3

First, read the performance tips: Performance Tips · The Julia Language

In particular, don’t use global variables. This can largely be achieved by 1) putting you code in a function or 2) prepending const = to you lines in data.jl.

You should also benchmark your code to identify where the slow points are.

If you data is sparse, JuMP uses a standard dictionary as the backing data structure anyway.

In order to get specific feedback, you should distill a small, minimal working example that demonstrates your code structure.

As one simplification:

if haskey(foo, key)
    foo[key]
else
   bar
end

# use

get(foo, key, bar)