Efficient way to get the JuMP.value of all variables in a JuMP.Model?

Specific Domains Optimization (Mathematical)
1

I wrote a method to get all the variable values in a JuMP Model:


function get_results_dict(m::JuMP.Model)
    av = JuMP.all_variables(m);
    d = Dict()
    for v in av
        # special handling of time series, sub-indices, etc.
        # e.g. a variable could have three sub-indices in a JuMP.Containers
        v = string(v) # e.g.  "x[a,2,3]"
        k = Symbol(v[1:prevind(v, findfirst('[', v))[1]]) # :x
        vm = value.(m[k])
        d[k] = vm.data
        #  e.g. typeof(vm) is JuMP.Containers.SparseAxisArray{Float64, 3, Tuple{SubString{String}, Int64, Int64}}
    end
    return d
end

However, this is extremely slow because

julia> length(av)
1349040

Is there more efficient way to get all the variable values? Other than only getting some variable values? And if the latter is the only solution, what is the most efficient way to get a group of variable values with different indexing structures in JuMP.Containers? (Say if I had a vector of variable names that is a subset of JuMP.all_variables).

2

Do you just want to get the variable values that are registered in object_dictionary(model)?

So something like the following:

d = Dict(
    k => value.(v) for 
    (k, v) in object_dictionary(model) if v isa AbstractArray{VariableRef}
)

You approach is very slow because it doesn’t short-circuit if the key is already found. Do something like:

function get_results_dict(m::JuMP.Model)
    av = JuMP.all_variables(m);
    d = Dict()
    for v in av
        # special handling of time series, sub-indices, etc.
        # e.g. a variable could have three sub-indices in a JuMP.Containers
        v = string(v) # e.g.  "x[a,2,3]"
        k = Symbol(v[1:prevind(v, findfirst('[', v))[1]]) # :x
        if !haskey(d, k)
            vm = value.(m[k])
            d[k] = vm.data
        end
        #  e.g. typeof(vm) is JuMP.Containers.SparseAxisArray{Float64, 3, Tuple{SubString{String}, Int64, Int64}}
    end
    return d
end
1 Like
3

I do have the short-circuit handling that you mentioned, specifically:

if k ∉ keys(d)

in place of your

if !haskey(d,k)

Maybe !haskey is much better than ∉?

Nonetheless, the object_dictionary method looks promising. But I’d like to handle special sub-indexing. For example:

julia> d = Dict(
           k => value.(v) for 
           (k, v) in object_dictionary(model) if v isa AbstractArray{VariableRef}
       )
Dict{Symbol, JuMP.Containers.SparseAxisArray{Float64, 3, K} where K<:Tuple{Any, Any, Any}} with 5 entries:
  :Qⱼ    =>   [632, 2, 4216 ]  =  0.0…
  :Pᵢⱼ   =>   [632-645, 3, 367  ]  =  0.0…
  :vsqrd =>   [632, 2, 4216 ]  =  0.997919…
  :Pⱼ    =>   [632, 2, 4216 ]  =  0.0…
  :Qᵢⱼ   =>   [632-645, 3, 367  ]  =  0.0…

julia> d[:Qⱼ]
JuMP.Containers.SparseAxisArray{Float64, 3, Tuple{SubString{String}, Int64, Int64}} with 280320 entries:
  [611, 3, 2821 ]  =  -0.0013604
  [632, 1, 2220 ]  =  0.0
  [632, 1, 7419 ]  =  0.0
  [632, 2, 3126 ]  =  0.0
...

I’d like to access the sub-indices of :Qⱼ like d[:Qⱼ]["611"][3][2821] (the indices are bus-name, phase, and time step in a power flow model). Is there something that I can do efficiently based on the typeof a value in object_dictionary, e.g. SparseAxisArray{Float64, 3, Tuple{SubString{String}, Int64, Int64}}?

(Edit: I’d like this indexing because then I can use the attributes of any given dimension for post-processing; sort of going from flat-file style single key-value pairs to sub-dictionary style indexing).

4

I do have the short-circuit handling that you mentioned

I don’t see it in your example?

Nonetheless, the object_dictionary method looks promising

I think what you really need is a proper data structure to handle the variables. Don’t dump everything into a JuMP model and then try to rediscover the structure after the fact.

But I’d like to handle special sub-indexing … like d[:Qⱼ]["611"][3][2821]

Then SparseAxisArray is not the type for you. If you want nested dictionaries, you’ll have to write that conversion yourself.

I will say that recent JuMP version support slicing like d[:Qⱼ]["611", :, :] but that creates a new dictionary each time which might be expensive.

5

If you want nested dictionaries, you’ll have to write that conversion yourself.

That is what I have done, and it’s really slow. So I’m asking how I do that efficiently.

So is the lesson to not use

@variable(model, x[dims1, dims2, dims3] >= 0)

where the dims1/2/3 are vectors of indices?

Maybe I should create dictionaries of variables and use the keys of those dictionaries to get the values? (But doesn’t that take away the efficiency of the object_dictionary?)

6

That’s great! This capability is one of the use-cases that I am trying to accomplish with my variable values sub-dictionaries, For example, what I want is

d[:Qⱼ]["611"][1] .+ d[:Qⱼ]["611"][2]

but now I can use

d[:Qⱼ]["611", 1, :] .+ d[:Qⱼ]["611", 2, :]

Is this capability released or do I need to dev JuMP to get it?

7

Is the solution. There are some nuances to how you define your variables though. If you do not want to use JuMP.Containers in the Dict values then you need to define variables in your own containers.

1 Like
8

So is the lesson to not use

@variable(model, x[dims1, dims2, dims3] >= 0)

where the dims1/2/3 are vectors of indices?

No, I don’t think that’s the lesson. I think the lesson is to choose an appropriate data structure for what you want to achieve. JuMP provides a few defaults, but they aren’t always the best choice:

In your case, you wanted x[i][j][k], which none of the JuMP containers support. So don’t try to force a JuMP container to meet your requirements; choose a different data structure up front.

do I need to dev JuMP to get it?

It was released in JuMP v1.3.0: Release notes · JuMP

If you do not want to use JuMP.Containers in the Dict values then you need to define variables in your own containers.

Sure. You could write a different function to value.(v) that handled things appropriately.

1 Like