[ANN] Upcoming breaking changes to CPLEX.jl and Gurobi.jl

Update: 20 October 2020: New versions of CPLEX.jl and Gurobi.jl are tagged.

tl;dr: CPLEX.jl and Gurobi.jl will have major breaking changes in the next latest release. If you just use CPLEX.Optimizer() or Gurobi.Optimizer() with JuMP, nothing changes. If you use any lower-level functionality in CPLEX.jl or Gurobi.jl (e.g., solver-dependent callbacks, or functions from the C API), your code will break. Furthermore, all support for MathProgBase has been removed. You can try out the changes by adding the master branch of each package. We will wait at least two weeks before tagging to give time for feedback. If no issues are raised during that time, we will tag new releases on October 20.

Dear all,

We have some big breaking changes landing in the next releases of CPLEX.jl and Gurobi.jl wrappers.

All low-level functions in the wrapper have been removed and replaced by a complete rewrite using Clang.jl. If you used any of these functions, your code will break. In addition, all support for MathProgBase has been removed.

Why did we do this?

CPLEX and Gurobi are two of the oldest solver wrappers, and the C wrappers have evolved on an ad-hoc basis over time. (The first commit with working code was made to Gurobi.jl on March 9, 2013!) This meant that we had frequent issues with “missing” functions in the C API that were not wrapped.

The age of the code also meant that the naming scheme was inconsistent. A C function like GRBaddqpterms might be called add_qpterms!, add_qp_terms, or c_api_addqpterms. In addition, the arguments to the functions were often different to the underlying C API and some arguments were not exposed. Combined with the lack of documentation, this meant it was hard to identify which Julia function matched which C function, and which arguments to supply.

The main reason you would want to use the C API is in a solver-dependent callback. In addition to now supporting the full functionality of a solver-dependent callback, we now have documentation. See the solver-dependent callbacks section below.

What did we do?

We used Clang.jl to wrap the complete C API of each solver.

The functions now have a consistent naming scheme, the arguments are all available and consistent, and the complete API is already documented by each solver.

In addition, all support for MathProgBase has been removed.

How do I try it out?

To try out the latest changes, use the following:

import Pkg
Pkg.add(Pkg.PackageSpec(name = "CPLEX", rev = "master"))
# or
Pkg.add(Pkg.PackageSpec(name = "Gurobi", rev = "master"))

Then restart Julia for the change to take effect.

If your code breaks, and you want to revert to the old API, use:

import Pkg
Pkg.add(Pkg.PackageSpec(name = "CPLEX", version = v"0.6"))
# or
Pkg.add(Pkg.PackageSpec(name = "Gurobi", version = v"0.8"))

Then restart Julia for the change to take effect.

If you have comments or feedback, please open an issue:

When will new versions be tagged?

We will wait at least two weeks before tagging new versions to give people time to try out the new versions and provide feedback. If no substantive issues are raised, we will tag new releases on October 20.

CPLEX Changes

New minimum version requirement: CPLEX.jl now requires CPLEX version 12.10.

CPLEX has extensive documentation for their C API.

The main changes are:

  • Function names have changed. For example, CPLEX.close_CPLEX(env) is now CPXcloseCPLEX(env).
  • For users of CPLEX.Optimizer, model.inner has been replaced by the fields model.env and model.lp, which correspond to the environment and problem pointers at the C API level. For example:
    model = direct_model(CPLEX.Optimizer())
    cpx_model = backend(model)
    stat = CPLEX.get_status(cpx_model.inner)
    
    is now:
    model = direct_model(CPLEX.Optimizer())
    cpx_model = backend(model)
    stat = CPXgetstat(cpx_model.env, cpx_model.lp)
    
  • Querying functionality has changed. For example:
    is_point = CPLEX.cbcandidateispoint(cb_data)
    
    is now:
    is_point_P = Ref{Cint}()
    ret = CPXcallbackcandidateispoint(cb_data, is_point_P)
    if ret != 0
        # Do something because the call failed
    end
    is_point = is_point_P[]
    

Gurobi changes

New minimum version requirement: Gurobi.jl now requires Gurobi version 9.0 or higher.

Gurobi has extensive documentation for their C API.

The main changes are:

  • Constants have changed. For example CB_MIPNODE is now GRB_CB_MIPNODE to match the C API.
  • Function names have changed. For example free_env(env) is now GRBfreeenv(env).
  • For users of Gurobi.Optimizer(), model.inner is now a pointer to the C model, instead of a Gurobi.Model object. However, conversion means that you should always pass model instead of model.inner to the low-level functions. For example:
    model = direct_model(Gurobi.Optimizer())
    grb_model = backend(model)  # grb_model is Gurobi.Optimizer
    # Old
    Gurobi.tune_model(grb_model.inner)
    # New
    GRBtunemodel(grb_model)
    
  • Some functions have been removed entirely. For example:
    using JuMP, Gurobi
    model = direct_model(Gurobi.Optimizer())
    optimize!(model)
    grb_model = backend(model)
    stat = Gurobi.get_status_code(grb_model.inner)
    
    is now:
    using JuMP, Gurobi
    model = direct_model(Gurobi.Optimizer())
    optimize!(model)
    valueP = Ref{Cint}()
    grb_model = backend(model)
    ret = GRBgetintattr(grb_model, "Status", valueP)
    if ret != 0
        # Do something because the call failed
    end
    stat = valueP[]
    

Solver-dependent callbacks

Here is an example with the latest syntax for CPLEX:

using JuMP, CPLEX, Test

model = direct_model(CPLEX.Optimizer())
set_silent(model)

# This is very, very important!!! Only use callbacks in single-threaded mode.
MOI.set(model, MOI.NumberOfThreads(), 1)

@variable(model, 0 <= x <= 2.5, Int)
@variable(model, 0 <= y <= 2.5, Int)
@objective(model, Max, y)
cb_calls = Clong[]
function my_callback_function(cb_data::CPLEX.CallbackContext, context_id::Clong)
    # You can reference variables outside the function as normal
    push!(cb_calls, context_id)
    # You can select where the callback is run
    if context_id != CPX_CALLBACKCONTEXT_CANDIDATE
        return
    end
    # You can query CALLBACKINFO items
    valueP = Ref{Cdouble}()
    ret = CPXcallbackgetinfodbl(cb_data, CPXCALLBACKINFO_BEST_BND, valueP)
    @info "Best bound is currently: $(valueP[])"
    # As well as any other C API
    x_p = Vector{Cdouble}(undef, 2)
    obj_p = Ref{Cdouble}()
    ret = CPXcallbackgetincumbent(cb_data, x_p, 0, 1, obj_p)
    if ret == 0
        @info "Objective incumbent is: $(obj_p[])"
        @info "Incumbent solution is: $(x_p)"
        # Use CPLEX.column to map between variable references and the 1-based
        # column.
        x_col = CPLEX.column(cb_data, index(x))
        @info "x = $(x_p[x_col])"
    else
        # Unable to query incumbent.
    end

    # Before querying `callback_value`, you must call:
    CPLEX.load_callback_variable_primal(cb_data, context_id)
    x_val = callback_value(cb_data, x)
    y_val = callback_value(cb_data, y)
    # You can submit solver-independent MathOptInterface attributes such as
    # lazy constraints, user-cuts, and heuristic solutions.
    if y_val - x_val > 1 + 1e-6
        con = @build_constraint(y - x <= 1)
        MOI.submit(model, MOI.LazyConstraint(cb_data), con)
    elseif y_val + x_val > 3 + 1e-6
        con = @build_constraint(y + x <= 3)
        MOI.submit(model, MOI.LazyConstraint(cb_data), con)
    end
end
MOI.set(model, CPLEX.CallbackFunction(), my_callback_function)
optimize!(model)
@test termination_status(model) == MOI.OPTIMAL
@test primal_status(model) == MOI.FEASIBLE_POINT
@test value(x) == 1
@test value(y) == 2

Here is an example with the latest syntax for Gurobi:

using JuMP, Gurobi, Test

model = direct_model(Gurobi.Optimizer())
@variable(model, 0 <= x <= 2.5, Int)
@variable(model, 0 <= y <= 2.5, Int)
@objective(model, Max, y)
cb_calls = Cint[]
function my_callback_function(cb_data, cb_where::Cint)
    # You can reference variables outside the function as normal
    push!(cb_calls, cb_where)
    # You can select where the callback is run
    if cb_where != GRB_CB_MIPSOL && cb_where != GRB_CB_MIPNODE
        return
    end
    # You can query a callback attribute using GRBcbget
    if cb_where == GRB_CB_MIPNODE
        resultP = Ref{Cint}()
        GRBcbget(cb_data, cb_where, GRB_CB_MIPNODE_STATUS, resultP)
        if resultP[] != GRB_OPTIMAL
            return  # Solution is something other than optimal.
        end
    end
    # Before querying `callback_value`, you must call:
    Gurobi.load_callback_variable_primal(cb_data, cb_where)
    x_val = callback_value(cb_data, x)
    y_val = callback_value(cb_data, y)
    # You can submit solver-independent MathOptInterface attributes such as
    # lazy constraints, user-cuts, and heuristic solutions.
    if y_val - x_val > 1 + 1e-6
        con = @build_constraint(y - x <= 1)
        MOI.submit(model, MOI.LazyConstraint(cb_data), con)
    elseif y_val + x_val > 3 + 1e-6
        con = @build_constraint(y + x <= 3)
        MOI.submit(model, MOI.LazyConstraint(cb_data), con)
    end
end
# You _must_ set this parameter if using lazy constraints.
MOI.set(model, MOI.RawParameter("LazyConstraints"), 1)
MOI.set(model, Gurobi.CallbackFunction(), my_callback_function)
optimize!(model)
@test termination_status(model) == MOI.OPTIMAL
@test primal_status(model) == MOI.FEASIBLE_POINT
@test value(x) == 1
@test value(y) == 2

Other changes

The upcoming releases also contain numerous improvements to the MathOptInterface wrappers of both solvers. For example, Gurobi gained support for indicator constraints and a solve can now be interrupted via CTRL+C. In CPLEX, binary variables without explicit bounds no longer throw a warning, and dual variables are returned for quadratic constraints.

Thanks,
Oscar

19 Likes

For me (Julia beginner, AMPL oldtimer) it is not clear how to use
the normal cplex routines:

using JuMP, XLSX, CPLEX

m = Model(CPLEX.Optimizer)

CPXwriteprob( m.env, m.lp, "mymodel", "LP" )
CPXXsetlogfilename( m.env, "mylogfile.txt", "a") 
CPXrefineconflict( m.env, m.lp, NULL, NULL);

I get an error:
ERROR: LoadError: type Model has no field env

I use CPLEX 12.10 and CPLEX.jl 0.6.6

Would be great if you can tell me how to call the Cplex routines.
Best greetings Gerhard

For me (Julia beginner, AMPL oldtimer) it is not clear how to use the normal cplex routines:

These routines are for advanced users only. It is expected you have a good understanding of the CPLEX C API.

I have updated the post above to clarify you need to use

model = direct_model(CPLEX.Optimizer())
m = backend(model)

instead of

m = Model(CPLEX.Optimizer)

Also note that NULL is C_NULL, and these function are only available on CPLEX#master, not 0.6.6.

2 Likes

Thank you very much for the help!

It worked as you described with

model = direct_model(CPLEX.Optimizer())
cpx_model = backend(model)
CPXsetlogfilename(cpx_model.env, "mylogfile.txt", "a")
stat = CPXgetstat(cpx_model.env, cpx_model.lp)

but with

m = Model(CPLEX.Optimizer)

it did not work.
ERROR: type CachingOptimizer has no field env

So just for completeness: Is there a way to access CPLEX routines also in CachingOptimizer - Model(CPLEX.Optimizer)?

Thank you very much for the help!
Please also add this info given above (on how to use CPLEX routines), also to the cplex.jl documentation. The use of solver routines is essential e.g. for trouble shooting an infeasible problem using IIS. Best Greetings Gerhard

You can use backend(model).optimizer.model.env and backend(model).optimizer.model.lp, but I don’t recommend this, nor will we document it, since it relies on internal JuMP representations. Use direct_model instead.

The use of solver routines is essential e.g. for trouble shooting an infeasible problem using IIS.

JuMP has documented support for accessing the IIS: https://jump.dev/JuMP.jl/stable/solutions/#Conflicts-1. The main reason to use the C API is for solver-specific callbacks.

Following up on this, new releases for both packages are now tagged and available in the General registry.

The latest versions are:

  • CPLEX.jl v0.7.0
  • Gurobi.jl v0.9.0

Please open an issue if you experience issues.

1 Like