JuMP: Behaviour of value for infeasible models

Hello everyone,

I stumbled on a potential bug in JuMP and its dependencies, however, I wanted to ask this question here first as I am unsure whether this intended behavior and (if not) on which library level this should be repaired.

For reference, I used the following library versions:
GLPK v1.1.3
Gurobi v1.0.2
JuMP v1.19.0
(all examples below assume using JuMP, GLPK, Gurobi)

I observed that there are cases where calling value(x) for a variable x of an infeasible model will not throw an error but return some value.
For example, in the case of GLPK this behaviour can be triggered as follows:

model = Model(GLPK.Optimizer)
# Create variable x in [-1,1]
@variable(model, -1 <= x <= 1)
# Add constraint that x>=2 (Problem is thus infeasible)
@constraint(model, x >= 2)
optimize!(model)
@assert termination_status(model) == MOI.INFEASIBLE
# This should throw an error but instead returns an error
println(value(x))

Obviously, this mistake can be avoided by first checking has_values(model) which correctly tells me that now model assignment exists, but to me it feels wrong that value returns something other than an error – especially given value is only supposed to provide primal values.

We can compare this to the behavior of Gurobi on the same problem which produces an error for the same problem:

model = Model(Gurobi.Optimizer)
@variable(model, -1 <= x <= 1)
@constraint(model, x >= 2)
optimize!(model)
@assert termination_status(model) == MOI.INFEASIBLE
# This throws an error:
println(value(x))

At first, I thought this was therefore only a problem with GLPK, but then I managed to track it to the different outcomes for MOI.get(model, MOI.ResultCount()) between the two models: GLPK returns 1 for this call as it has an infeasibility certificate.
Since MOI.check_result_index_bounds uses MOI.ResultCount() to check if there exists a result, this function does not throw an error and makes it so that value returns something.

On the other hand, MOI.get(model, MOI.ResultCount()) returns 0 for Gurobi (this problem was solved by Gurobi’s presolve without an infeasibility certificate) and consequently MOI.check_result_index_bounds throws an error.

As a consequence, when we deactivate Gurobi’s presolve (thus ensuring that Gurobi has an infeasibilty certificate) we also get a value from Gurobi:

model = Model(Gurobi.Optimizer)
# Deactivate Presolve -> ensures we get an infeasibility certificate
set_attribute(model, "Presolve", 0)
@variable(model, -1 <= x <= 1)
@constraint(model, x >= 2)
optimize!(model)
@assert termination_status(model) == MOI.INFEASIBLE
# This no longer throws an error:
println(value(x))
# Because we now have a positive result count:
@assert MOI.get(model, MOI.ResultCount()) >0
@assert model.moi_backend.optimizer.model.has_infeasibility_cert

This leaves me with the following questions:
Should value be returning values for variables when the problem is infeasible?

If yes: What is the meaning of those values?

If no: On which level should this be fixed?

  • MOI’s check_result_index_bounds currently calls get(model, ResultCount()) maybe there’s a better way to perform this check?
  • One could fix . the invocations of check_result_index_bounds in the individual solver libraries, but this seems like a lot of changes would be necessary and I am not sure which other libraries might have the same issue
  • One could introduce a check earlier up in the chain (e.g. checking for has_values upon an invocation of value

So far I’ve only been a JuMP user, so I have not that much knowledge about the inner workings of JuMP. Thus, I thought it best to ask on here which option (if any) is the best way forward.

Thanks in advance,

Samuel

Hi @samysweb, welcome to the forum.

This design is intentional.

It is not safe to rely on value throwing an error. To understand the solution returned by the solver you should check termination_status(model), primal_status(model), and dual_status(model).

Take a look at this:

julia> using JuMP, GLPK, HiGHS, Gurobi

julia> function main(optimizer)
           model = Model(optimizer)
           set_silent(model)
           @variable(model, -1 <= x <= 1)
           @constraint(model, x >= 2)
           optimize!(model)
           println("Solver             : ", optimizer)
           println("termination status : ", termination_status(model))
           println("result count       : ", result_count(model))
           println("primal status      : ", primal_status(model))
           println("dual status        : ", dual_status(model))
           println()
       end
main (generic function with 3 methods)

julia> for optimizer in (GLPK.Optimizer, HiGHS.Optimizer, Gurobi.Optimizer)
           main(optimizer)
       end
Solver             : GLPK.Optimizer
termination status : INFEASIBLE
result count       : 1
primal status      : NO_SOLUTION
dual status        : INFEASIBILITY_CERTIFICATE

Solver             : HiGHS.Optimizer
termination status : INFEASIBLE
result count       : 0
primal status      : NO_SOLUTION
dual status        : NO_SOLUTION

Solver             : Gurobi.Optimizer
termination status : INFEASIBLE
result count       : 0
primal status      : NO_SOLUTION
dual status        : NO_SOLUTION

GLPK reports that it has one result because it found an infeasibility certificate. Gurobi and HiGHS did not find an infeasibility certificate (likely because they proved infeasibility during presolve).

See Solutions · JuMP for a recommended workflow.

2 Likes

Hi @odow ,

thanks a lot for the welcome and the detailed reply.

I tend to disagree with this design decision, as it seems like the kind of thing many people will get unintentionally wrong (maybe even by just forgetting the terminaiton_status check) – especially since it’s not clear to me whether the return value of value(x) has any (interesting) meaning in the case of NO_SOLUTION?
On the other hand, changing this now could be considered a breaking change in the interface…

Going forward, I for my part will make sure always to check the solution status beforehand.

Best and thanks again,

Samuel

as it seems like the kind of thing many people will get unintentionally wrong (maybe even by just forgetting the termination_status check)

This was one consideration. We think people must always check the status of the solver to see what solution was returned.

has any (interesting) meaning in the case of NO_SOLUTION ?

It shouldn’t. And I agree, if we could do things differently, this might be a candidate for throwing an error.

But, if we threw an error only if NO_SOLUTION was returned, then people might rely on “if it didn’t error then it’s okay.” But solvers can return infeasible points, nearly feasible points, or certificates, so returning a value does not mean that the value is good.

Even if there is a FEASIBLE_POINT returned, it does not mean that the point is optimal. The solver could have stopped for any number of reasons, and only termination_status can tell you why.

This question does come up pretty frequently though, so perhaps I need to improve the documentation. (Edit: okay. We don’t follow our own recommendations much in the tutorials. I’m working on a PR: [docs] check termination and primal status after optimize! by odow · Pull Request #3668 · jump-dev/JuMP.jl · GitHub)

2 Likes

I’d appreciate any thoughts on this: [docs] check termination and primal status after optimize! by odow · Pull Request #3668 · jump-dev/JuMP.jl · GitHub

Is it enough for us to just add a bunch of assertions to the documentation?

Do people understand the difference between OPTIMAL and LOCALLY_SOLVED?

Would it help if we added a function like this:

"""
    has_solution(model::Model; dual::Bool = false)

Return `true` if the model has a feasible primal solution to return and the
solver terminated normally. If `dual`, additionally check that a feasible
dual solution is available.

If this function returns `false`, use [`termination_status`](@ref), [`result_count`](@ref),
[`primal_status`](@ref) and [`dual_status`](@ref) to understand what solutions
are available (if any).
"""
function has_solution(model::Model; dual::Bool = false)
    ret = termination_status(model) in (OPTIMAL, LOCALLY_SOLVED) &&
          primal_status(model) == FEASIBLE_POINT
    if dual
        ret &= dual_status(model) == FEASIBLE_POINT
    end
    return ret
end
2 Likes

I think because assertions can be disabled it would be best to do error("...") in the docs.

Maybe point them to this explanation? Although I would add more details on the meaning of globally versus locally optimal. And perhaps a note on the kinds of problems where we typically get local solutions (for instance in LPs it’s rather rare).

YES!

1 Like

Thanks a lot for your work on making this easier to grasp, from my point of view this extension of the documentation would very much put a focus on this subtlety.

I think this would certainly help.
If this were to be added, I wonder if it makes sense to have an additional optional paramter to choose :optimal, :local or :any? (Or a more JuMP-idiomatic version of this…)
This would also make it very easy to check if there exists an optimal primal solution:

if has_solution(model,status=:optimal)
  print("We found an optimal solution!")
end

My reaction is not to add :optimal or :local, etc. The existing statuses communicate a wealth of information, and by the time you’re digging in to different types of solutions, you probably understand the differences and could easily just write termination_status(model) == LOCALLY_SOLVED.

I think this is really targeting newer JuMP users who may otherwise not check statuses.

It would be non-breaking to add such an option in the future though. So perhaps we could just start with has_solution(model; dual), then then see if we need something like global::Bool = false.

The other thing is to bike shed the name of has_solution. If the solver stopped due to a time limit, the return would be false, even if the solution might have a solution.

Or perhaps it should be has_optimal_solution and we don’t distinguish between globally and locally optimal. (or perhaps also include the global keyword).

Let me edit the PR, and we can work through some options.

I’m ashamed to admit that I don’t always check status either

Maybe is_success or sth like that?

See [docs] check termination and primal status after optimize! by odow · Pull Request #3668 · jump-dev/JuMP.jl · GitHub. I’ve gone with:

    has_optimal_solution(
        model::GenericModel;
        dual::Bool = false,
        allow_local::Bool = true,
        allow_almost::Bool = false,
        result::Int = 1,
    )

We’re still debating names (has_optimal_solution, is_optimize_success, is_solved_and_feasible…) so if anyone has a suggestion for a good name, please chime in: Add is_solved_and_feasible by odow · Pull Request #3668 · jump-dev/JuMP.jl · GitHub :smile: