Solver-dependent callbacks in JuMP -- How to do it right?

Inspired by this answer, I tried to create a small example using the JuMP direct_model together with solver-specific callbacks. I adapted the knapsack example provided with JuMP to reject solutions using the most efficient item (in CPLEX, incumbent solutions may be rejected without the need to add a Lazy Constraint that invalidates the solution, so, I am using Lazy Constraints without the constraints themselves in the example). The relevant CPLEX documentations are about the context, and the method to reject candidate.

I am aware there is an effort to make some generic callbacks to work (documented here), but while they are a nice facility for some specific cases, I prefer something that works in all cases (no matter how obscure is the solver-specific behavior). And I am also aware of the existence of this answer, but I am not sure if it is the same thing. In the mentioned answer, the MOI.get method is used, I am not sure if this is needed, and if this allows solver-specific behavior as the one I described for CPLEX.

The monster I assembled follows:

using JuMP, CPLEX, Test
const MOI = JuMP.MathOptInterface

function example_knapsack(; verbose = true)
    n = 5
    profit = [5, 3, 2, 7, 4]
    weight = [2, 8, 4, 2, 5]
    @test length(profit) == n
    @test length(weight) == n
    capacity = 10
    cplex_backend = CPLEX.Optimizer()
    model = direct_model(cplex_backend)
    @test backend(model) === cplex_backend
    @show typeof(cplex_backend)
    MOI.set(cplex_backend, MOI.NumberOfThreads(), 1)
    function my_callback(cb_context::CPLEX.CallbackContext, context_id::Clong)
        if context_id == CPLEX.CPX_CALLBACKCONTEXT_CANDIDATE
            primal_sol = Vector{Float64}(undef, n)
            objective = Ref{Float64}(0.0)
            CPLEX.cbcandidateispoint(cb_context) == 0 && return
            CPLEX.cbgetcandidatepoint(
                cb_context, primal_sol, Cint(0), Cint(n - 1), objective
            )
            # Just for testing the lazy constraints, here we reject
            # any solution using the fist variable of greatest efficiency
            # (which probably appears in the optimal solution of the model
            # without the lazy cuts). No constraint is actually added,
            # just the solution rejected.
            e = profit ./ weight
            max_e, idx = findmax(e)
            #@show max_e
            #@show idx
            #@show primal_sol[idx]
            if primal_sol[idx] ≈ 0
                println("accepted solution of value $(objective[])")
                @show primal_sol
            else
                CPLEX.cbrejectcandidate(
                    cb_context, # one of the callback parameters
                    Cint(0), # rcnt: number of constraints to add
                    Cint(0), # nzcnt: number of nonzeros of added constraints
                    Cdouble[], # rhs: righthand sides for the constraints
                    Cchar[], # sense: sense for the constraints
                    Cint[], # rmatbeg: indexes of the two arrays below where
                    # each constraint start (the two arrays below may have a
                    # variable number of elements for each constraint)
                    Cint[], # rmatind: indexes of nonzero columns 
                    Cdouble[] # rmatval: coefficients of nonzero columns
                )
                println("rejected solution of value $(objective[])")
                @show primal_sol
            end
        else
            error("Callback shold not be called from context_id $(context_id).")
        end
    end
    CPLEX.cbsetfunc(cplex_backend.inner, CPLEX.CPX_CALLBACKCONTEXT_CANDIDATE, my_callback)
    @variable(model, x[1:5], Bin)
    # Objective: maximize profit
    @objective(model, Max, profit' * x)
    # Constraint: can carry all
    @constraint(model, weight' * x <= capacity)
    # Solve problem using MIP solver
    JuMP.optimize!(model)
    if verbose
        println("Objective is: ", JuMP.objective_value(model))
        println("Solution is:")
        for i in 1:5
            print("x[$i] = ", JuMP.value(x[i]))
            println(", p[$i]/w[$i] = ", profit[i] / weight[i])
        end
    end
    @test JuMP.termination_status(model) == MOI.OPTIMAL
    @test JuMP.primal_status(model) == MOI.FEASIBLE_POINT
    #@test JuMP.objective_value(model) == 16.0
end

example_knapsack(verbose = true)

My concern is if this is the “right” way to do it. Should I use the MOI.get methods whenever possible? (I have some trouble knowing where to look for their documentation.) I needed to disable multithread to get this working (otherwise I got segmentation faults), there are other pitfalls? What is the best way to link the variables created by JuMP with their variable index in CPLEX? (I need to be sure I am referring to the right variables inside the callback)

Solver-specific does not mean that you should use the solver-specific API inside the callback. Because then you don’t know which CPLEX column index correspond to which JuMP variable.
Solver-specific means that you use MOI attributes defined in CPLEX such as CPLEX.CallbackFunction.
So instead of CPLEX.cbsetfunc, do MOI.set(model, CPLEX.CallbackFunction(), my_callback).
Inside the solver-dependent callback, you can use solver-independent attributes:

  • Instead of CPLEX.cbgetcandidatepoint, do MOI.get(model, MOI.CallbackVariablePrimal(cb_context), x[1]) where x[1] is a JuMP variable.
  • Instead of CPLEX.cbrejectcandidate, use MOI.get(model, MOI.LazyConstraint(cb_context), ...)

For more details, see example_solver_dependent_callback in https://github.com/JuliaOpt/JuMP.jl/blob/master/examples/callbacks.jl.
Note that before we release JuMP v0.21, you need JuMP master for this to work and before we release MOI v0.9.8, you need MOI master if you don’t use direct_model (with direct_model, MOI v0.9.7 is enough).

2 Likes

See the following. The file to look at for inspiration is https://github.com/JuliaOpt/CPLEX.jl/blob/44c8ca62b4e0e63e85a66e029e332d9dafeebd32/src/MOI/MOI_callbacks.jl. You need disable multi-threading because CPLEX calls the callback from any thread, and Julia’s C-interop is not thread safe.

using CPLEX
using JuMP
using Test

function example_knapsack()
    n = 5
    profit = [5, 3, 2, 7, 4]
    weight = [2, 8, 4, 2, 5]
    e = profit ./ weight
    max_e, idx = findmax(e)
    capacity = 10
    
    model = direct_model(CPLEX.Optimizer())
    MOI.set(model, MOI.NumberOfThreads(), 1)

    @variable(model, x[1:5], Bin)
    @objective(model, Max, profit' * x)
    @constraint(model, weight' * x <= capacity)

    function my_callback(cb_context::CPLEX.CallbackContext, context_id::Clong)
        if context_id != CPLEX.CPX_CALLBACKCONTEXT_CANDIDATE
            return
        elseif CPLEX.cbcandidateispoint(cb_context) == 0
            return
        end
        CPLEX.callbackgetcandidatepoint(backend(model), cb_context, context_id)

        # If using JuMP master branch:
        # x_idx_val = callback_value(cb_context, x[idx])

        # If using latest JuMP release:
        x_idx_val = MOI.get(
            backend(model), 
            MOI.CallbackVariablePrimal(cb_context),
            index(x[idx])
        )

        if x_idx_val ≈ 0
            return
        end

        CPLEX.cbrejectcandidate(
            cb_context,
            Cint(0),
            Cint(0),
            Cdouble[],
            Cchar[],
            Cint[],
            Cint[],
            Cdouble[]
        )
    end
    MOI.set(model, CPLEX.CallbackFunction(), my_callback)
    optimize!(model)
    @test JuMP.termination_status(model) == MOI.OPTIMAL
    @test JuMP.primal_status(model) == MOI.FEASIBLE_POINT
end

example_knapsack()

1 Like

I am grateful for the attention and fast answers but I think I am getting some dissonant information here.

@blegat, you say I should use the MOI attributes, as without them I cannot guarantee I am changing the right variable, but you do not explicitly say I must use the MOI attributes, was this intentional? Because your advice on rejecting solutions does not seem right. I am almost sure it is not a MOI.get(model, MOI.LazyConstraint(cb_context), ...) call but a MOI.submit and this MOI.submit takes a constraint, and I want to reject the solution without adding a constraint. I downloaded the master version of JuMP and tried to tinker a little with MOI.submit, but it does not seem I could use it if I just want to reject the solution without adding a lazy constraint. While using the MOI attributes may appear to be higher level and more productive (and I would like to use them if they are the best way to define solver-dependent callbacks), I have two problems with the MOI attributes: (1) some productivity is lost by finding which exact MOI attribute I should be using, considering that what I already know is the name of a callback of a specific solver and it is easier then to find the correspondent low-level method; (2) in the example I presented (described above), it seems like MOI cannot do what I want, I want to write solver-independent callbacks in a way that will always be possible to access any solver-specific behavior, otherwise I would be either using 0.18.5 or waiting for the new generic callbacks.

@odow, Your code does what I want and is careful to use the right variable index. I just want to know if there are other things I should be aware of when mixing JuMP, MOI, and C-API-Wrappers as you are doing. I mean, I suppose this is a valid way of defining callbacks, right? (i.e., not a workaround that will stop working between minor version changes of the packages, that depends on implementation details) I just need to make sure I am using the right column index or there are other kinds of indexes and references I need to convert between JuMP and the low-level wrappers?

It’s by design and guaranteed that when you use direct_model, the indices of the variables in JuMP match the indices in the solver’s MOI wrapper. The solver’s MOI wrapper could add an extra level of indirection, but I don’t think CPLEX does.

Fully agreed that solver-dependent callbacks are poorly documented and incomplete at the moment. We’re lacking contributors in this area.

1 Like

it seems like MOI cannot do what I want, I want to write solver-independent callbacks in a way that will always be possible to access any solver-specific behavior, otherwise I would be either using 0.18.5 or waiting for the new generic callbacks.

Accessing solver-specific behavior was not possible in JuMP 0.18, so the current implementation is strictly better than 0.18.5. The generic callbacks are a replacement for the old behavior.

I mean, I suppose this is a valid way of defining callbacks, right?

The code I posted above is our current preferred way of defining solver dependent callbacks.

(i.e., not a workaround that will stop working between minor version changes of the packages, that depends on implementation details)

Until JuMP reaches v1.0, things can change between minor version changes. It’s likely that the low-level CPLEX functions will change in the future, because I want to re-write and clean up the package prior to JuMP 1.0.

However, it won’t change in the near term, and is likely to be a simple renaming for your purposes.

I just need to make sure I am using the right column index or there are other kinds of indexes and references I need to convert between JuMP and the low-level wrappers?

In general, you should use JuMP variables. If you need to use an MOI index, call JuMP.index(x). If you need a column, be very careful. It’s likely easier to make a PR to the solver to add the required functionality at the MOI level.

A good example of converting between MOI indices and solver columns is here:

Figuring out which functions to call and what arguments to pass will require you to read and understand the source code of the wrapper.

For CPLEX, GLPK, and Gurobi, the functions are all in MOI_callbacks.jl files. Those are the first places to look.

Fully agreed that solver-dependent callbacks are poorly documented and incomplete at the moment. We’re lacking contributors in this area.

:100:

1 Like

Thank you very much for your answers.

What could I do to help with this problem? It is possible that I will explore a benders decomposition using this feature, so it is adjacent to something I would do anyway.

Write tests, documentation, and examples, but it’s a case of implement as needed. We don’t have a roadmap or guideline for how things should work. The space is pretty open for changes.

It’s probably a case of you saying “I want to do X,” figuring out how to do it, making any required changes, and then documenting the process.

1 Like

Perfect. I will keep it in mind.