Why CPLEX’s Automatic Benders Outperforms Callback Benders in Julia?

Hi all,

I am trying to understand how to reproduce CPLEX’s automatic Benders decomposition using callbacks in Julia (via CPLEX.jl). CPLEX provides an automatic Benders mode, and the underlying procedure is described in the paper “Implementing Automatic Benders Decomposition in a Modern MIP Solver.”

In my experiments, I can partially replicate the in–out technique at the root node using callbacks, but I am not able to replicate CPLEX’s behavior deeper in the branch-and-bound tree.

I have two main questions:

  1. Multithreading and callbacks:
    CPLEX’s automatic Benders mode appears to generate cuts across multiple threads during the branch-and-bound search. If one tries to implement Benders via user callbacks, multithreading is typically unsafe unless the solver guarantees thread safety. How does CPLEX handle this internally? Are they avoiding callbacks entirely, or do they have a thread-safe mechanism not exposed to users?

  2. Efficiency differences:
    When I disable all other MIP cuts and restrict CPLEX to a single thread, the automatic Benders decomposition still produces far fewer Benders cuts and solves significantly faster than my callback-based implementation. I am unsure why this occurs. In my callback implementation, the branch-and-bound tree explores many integer nodes for capacitated facility-location problems, and this leads to the generation of a very large number of Benders cuts. What aspects of CPLEX’s internal Benders implementation allow it to use fewer cuts and avoid exploring so many nodes?

I attached the example code below. Any insights into how CPLEX’s automatic Benders decomposition differs from what can be implemented through user callbacks would be greatly appreciated.

cplex_cflp.jl (4.3 KB)

Hi @ASUKaiwenFang,

Because CPLEX is proprietary software, we don’t know how they implement things internally. Unfortunately, that means you’re unlikely to get any concrete answers to your questions :cry:

Hi @odow ,

That’s true lol. Since I’m trying to develop a JuMP-based generic Benders decomposition library, I ran into these questions and became very curious about how CPLEX manages this internally.

I took a read of the abstract and introduction of that paper and I feel like I can safely skip that paper. (one very minor reason is that they only mentioned “CGLP”, but nowadays we can also generate cuts from MILP subproblems)

Benders decomposition is under the umbrella of decomposition techniques for large-scale structured MILPs. Decomposition techniques just depend very much on specific problems.

I think the direction is not very appealing. The adj. “Generic” means the user just voluntarily give up opportunities to maximize algorithm performance for a specific class of problems.

I took a read of the abstract and introduction of that paper and I feel like I can safely skip that paper. (one very minor reason is that they only mentioned “CGLP”, but nowadays we can also generate cuts from MILP subproblems)
Benders decomposition is under the umbrella of decomposition techniques for large-scale structured MILPs. Decomposition techniques just depend very much on specific problems.

Yes, although the paper focuses mainly on CGLP, I agree that decomposition techniques are highly problem-dependent and that exploiting structure can certainly lead to much stronger cuts. However, that does not imply that generic techniques are useless. Methods like inout technique and GBC, for example, remain broadly applicable across many problem classes.

The adj. “Generic” means the user just voluntarily give up opportunities to maximize algorithm performance for a specific class of problems.

Regarding the second point, you are absolutely right. “Generic” was not the most accurate word for what I intended. What I mean is that, in my research, I often work with multiple problems, multiple model formulations, multiple Benders cut generation schemes, and various solving workflows. At the moment, I haven’t found an existing library that can conveniently support this diversity.

My goal is to design a modular framework that can accommodate different problem structures and algorithmic variants, so that I can use one unified codebase to conduct comprehensive computational experiments. The intention is to achieve a balance between flexibility, ease of use, and computational performance, and to provide a platform that facilitates developing, analyzing, and benchmarking Benders decomposition–based algorithms.

Cool! A generic benders library for JuMP is often asked for. Do you take monolithic problem and split it apart or make the user build separate sub problems (a la SDDP.jl)?

I designed several approaches:

  1. Manual construction:
    Users can manually build the master and subproblems using macros, similar to the workflows in SDDP.jl and StochasticPrograms.jl.
  2. Automatic decomposition:
    There is also an automatic decomposition function that takes a JuMP model as input and returns the corresponding master problem and subproblem(s).
  3. Problem-specific customization:
    I provide an interface that allows users to define their own methods for constructing the master and subproblems, using multiple dispatch based on the problem type(data) and the chosen cut-generation strategy.

Nice! Both 1 and 2 have pros and cons

You’re just encouraging interface development, but @ASUKaiwenFang 's real concerns here are performance.

First, from my perspective, Benders decomposition aims to build a strong dual bound at the root node (i.e. before branch-and-bound is enacted). In other words, if I bother to use a Benders decomposition framework rather than solving a monolithic MIP, I wish I can get a narrow rgap at the root node before branch-and-bound happens.

If your Benders master problem is an MIP (e.g. a unit commitment problem), then typically you need at first generate trial points with an LP-relaxed master problem. In this phase you don’t need the callback APIs provided by CPLEX.

The branch-and-bound (combinatorial) phase is exactly why MIP is NP-hard. Therefore there are many issues at this phase, e.g. how should you generate cut, within a callback or not, at integer nodes or not… These things are just very tricky. I’ve discussed these issues with other researchers before.

Hi Walter, I’m going to ask that we keep discussions positive. Just because you don’t find a JuMP extension for Benders interesting doesn’t mean that others won’t. Kaiwen and I are both well aware of the numerical and algorithmic challenges with implementing Benders.

You’re just encouraging interface development, but @ASUKaiwenFang 's real concerns here are performance.

Both aspects are within the scope of what I am considering. Since there are not many publicly available reference implementations of Benders decomposition, I appreciate any suggestions or perspectives that others can share.

The branch-and-bound (combinatorial) phase is exactly why MIP is NP-hard. Therefore there are many issues at this phase, e.g. how should you generate cut, within a callback or not, at integer nodes or not… These things are just very tricky. I’ve discussed these issues with other researchers before.

You’re right — it is quite tricky. Even though there may be no universal criterion that works for all problems, if we can design modular interfaces to handle some techniques, users would be able to “play” with Benders decomposition like LEGO, assembling different components as needed.

There is one small issue if you add cuts to the master problem via callback functions:

  • Will those lazy cuts be added to the JuMP Model? e.g. if you had added 10 cuts, and the CPLEX is logging, but you interrupt it via ^C, so you return to julia REPL. You query the JuMP Model of the master problem, will those 10 lazy cuts be with that Model? (IIRC, this is not the case for Gurobi.)

(I recall this existing Benders’ interface Decomposition paradigms · Coluna.jl)

Will those lazy cuts be added to the JuMP Model?

Nope. If you want to store cuts from a callback, you must manually store them in a user-controlled data structure.

I think the Bonami, Salvagnin and Tramontani paper actually gives a fair amount of insight into this, even if we cannot know every detail of the current proprietary implementation. The implementation described there is not simply the same callback algorithm running in C. CPLEX identifies the decomposition, presolves the complete model, decomposes the presolved model, runs a stabilized Benders loop on the LP relaxation, and then starts branch-and-cut with Benders cuts separated as lazy constraints. Full model presolve and native access to LP bases and cut management are difficult to reproduce when the master and worker are constructed separately.

Walter’s point about the root bound is therefore where I would start. If the callback implementation enters the tree with a weaker bound than CPLEX obtains after its initial Benders loop, the larger tree is not surprising. I would compare the final root bounds and the time and number of cuts needed to reach them before comparing the rest of the search.

There is also a paper that is especially relevant to this example. Fischetti, Ljubić and Sinnl describe a callback Benders implementation for essentially the same capacitated facility location problem in Benders decomposition without separability. It contains several refinements that seem more directly applicable than generic Benders advice.

One thing that is easy to miss in the posted formulation is that the aggregate capacity constraint already guarantees feasible recourse. Assignments are continuous and every customer can be served by every facility, so the worker should not generate feasibility cuts or Farkas rays. The relevant cut quality issue is the degeneracy of the transportation LP and the choice between alternative optimal dual solutions.

The CFL paper handles this by recomputing the optimality cut coefficients rather than simply accepting the reduced costs returned by the LP solver. It fixes the dual multipliers for the customer assignment constraints and obtains each facility coefficient from a continuous knapsack problem. These knapsacks are cheap to solve by sorting. This reduces the arbitrary coefficient choice caused by dual degeneracy and produced more stable cuts in their implementation. It seems a more targeted improvement for this problem than feasibility cut normalization.

They also keep the transportation worker persistent and solve it using dual simplex so that each solve reuses the previous basis. Their in-out procedure deliberately keeps successive separation points fairly close in the linear CFL case, making the basis warm start more effective. This is worth considering if the current in-out implementation chooses points without accounting for how much the worker changes between solves.

Their separation policy in the tree is also quite selective. They limit repeated separation at a node and stop calling the worker at fractional points when separation is taking too much time or is unlikely to prune the node. Integer candidates are still always checked through the lazy callback, so correctness is preserved. This directly addresses the problem described in the question, where the callback implementation generates a very large number of cuts.

Another useful idea is their restart procedure. After processing the root, they retain the useful callback cuts, add them to the master as ordinary constraints, and resolve the root before starting the final tree search. This allows presolve, variable fixing and CPLEX’s internal cuts to use the accumulated Benders information from the beginning. It also gives a practical reason to store callback cuts separately, as discussed earlier in the thread.

I would add y[i,j] <= x[i] as well. These inequalities are redundant when x is binary, but they strengthen the LP relaxation and give the generalized bound structure discussed in the CPLEX paper. In a persistent worker they can be handled as bound changes, provided their dual contribution is included when constructing the cut.

On multithreading, the limitation is not Benders itself. A native CPLEX callback can run during parallel MIP search if it is thread safe. The CFL implementation creates one persistent worker clone for each CPLEX thread, so simultaneous callback calls do not share a worker model or LP basis. A JuMP implementation would need the same architecture, together with CPLEX.jl support for callbacks arriving on multiple solver threads, thread local worker models and careful handling of shared cut state. CPLEX.jl currently documents its callbacks for single threaded use, so this cannot safely be reproduced merely by setting Threads > 1. It does not explain the single thread comparison here, but it does explain an architectural advantage of CPLEX’s internal implementation.

There are also a few benchmarking issues in the attached script. The seed argument is unused, the transportation cost matrix is transposed relative to its later indexing, and only cover, flow cover and MIR cuts are disabled rather than all other CPLEX cut families. These should be corrected before relying on the cut and node counts. CPXPARAM_Benders_Strategy = 1 is not itself a problem and is probably the fairest comparison when both methods use the same partition.

My reading is that the difference is likely to come from the combination of full model presolve, a stronger root phase, better treatment of degenerate optimality cuts, basis aware worker solves and more selective separation in the tree. It is not simply that CPLEX generates the same cuts faster.

It is true that the master cutting plane model will become harder to be re-solved (to OPTIMAL) when more cuts are appended. And it is true that the dual simplex algorithm should be employed here (Gurobi Method=1). To me, this is the major bottleneck (using multi epigraphical variable).

I think I’ve run out of novel research ideas on Benders Decomposition :neutral_face:, such that I’m a bit unclear what future topic I should go with.

On the other hand, however, I have no inclination to implement branch-and-bound in my own code manually —the engineering bar is too high for me. I found in my experiments that I can close the major optimality gap at the root node. The point of a cutting plane is to “mark” a (high-level) primal value at one “vertex” and at the same time to give a lower-bounding plane for the other region. The added cuts all together constitute a convex PWL lower bound. This usage in integer programming is appreciable.