In the last release we spent some time improving JuMP for use by AI agents. Error messages are now clearer and include an actionable next step, which helps agents debug issues. (This also helps regular users!) We added an AGENTS.md file, and we added learning intentions to the tutorials so that you can skim whether a tutorial is relevant before reading the entire thing.
Before we go further, we have some questions for the community.
How do you use AI with JuMP?
Do you see common places where agents get stuck?
Do you use custom skills?
Do you have suggestions for how we could improve AI integration with JuMP?
Let us know by commenting below, DM me privately, or open a GitHub issue.
Debugging slow convergence hasn’t been very successful. The default verbosity is perhaps not very AI-friendly, and certainly not token-efficient, so I instruct the agent to use set_silent unless it has a reason to do otherwise. I wonder if some other kind of summary would be better to help the AI diagnose convergence issues.
I guess this is hard because it is solver-dependent.
Relatedly: I have been on a rant at some solver developers recently. I don’t think their logs are helpful, and this is more evidence. The way solvers present logs has been largely guided by what solver developers like to see, not what is helpful to users. There’s also a strong push for backwards compatibility because many people have written scripts to parse log files and they don’t want to break them.
I think if we presented simpler logs in a more readable format, more people might actually pay attention to them instead of just turning them off.
Mainly as an auto-complete-on-steroids, that’s where my productivity is improved for real. My experience with vibe coding is quite poor honestly (except for generating tests).
Do you see common places where agents get stuck?
When it gets too mathematical. It seems to perform poorly when it comes to understanding the underlying optimization model (the quadratic model, the frigging \frac{1}{2} term in front of the Hessian, the sparsity pattern, the prediction equations of the linear optimal control problem, etc.)
Do you use custom skills?
No. I find skills to be too messy in terms of traceability, especially when there is a stack of them (“why this guardrail is suddenly activated?!?”)
Do you have suggestions for how we could improve AI integration with JuMP?
No. Except maybe a request: please do not use it too much for the implementation of new features in JuMP.jl. You are doing an incredible job since the beginning, please keep this marvellous pace (a human maintainer, the best kind ).
Thanks @odow for all your work in the maintenance of JuMP.jl and MathOptInterface.jl !
Oscar, thank you again for all the time you spend making our lives easier
How do you use AI with JuMP?
The list below might not be exhaustive, but I’ve gone through each bullet at least one.
Generate a mathematical formulation of a problem described natural-language.
(granted, this is not JuMP specific)
Given a mathematical model of an optimization problem, write code that implements said formulation. Can also go straight from natural language description to code, although I personally find it harder to validate the math.
Generate unit tests examples for a given problem / formulation / implementation.
Generate documentation for a given implementation. This includes generating docstrings as well as some higher-level documentation, for instance, writing the mathematical model given the code.
Some infra/scaffolding work that is JuMP-adjacent for which I use agents a lot
Generating data formats, parsers, types, etc… for manipulating the data I work with. This typically covers problem data, model/optimizer data (especially if I am coding my own algorithm), and solution data.
Building benchmarking infrastructure (parsing instances, collecting results, analyzing results and building graphs / tables). Again not 100% JuMP-specific but typically a huge time sink for me.
Use cases I can imagine but have never gone through myself
Diagnostics/validity – “The solver says my problem is infeasible but I know it’s not” or “The solver returns this solution but I know it shouldn’t be feasible”
Diagonstics/performance – “My problem is not solving as fast as I want, agent please help me”
Formulation improvement – “Please help me find a better formulation for this”
Do you see common places where agents get stuck?
I’ve seen a few cases where an agent / LLM would make a mathematical error, e.g., using duality concepts while working with a mixed-integer model, or small mistakes while writing a formulation (especially when dealing with a large context window / formulation).
I’m usually able to detect them at first sight, but ensuring mathematical validity is a huge concern to me in general (both for human and agent-generated code)
Big M formulations need a good choice of big M (not too small, not too large)… and agent-generated big Ms are often not that.
Complex indexing scheme: multiple dimensions with sparse axes increases the risk of a mistake.
This is not getting stuck per se, but I most often find agent-generated code cumbersome and hard to read / follow. Too many loops unrolled, too many things bundled together, etc…
This is something that would likely get fixed (to some extent) via a skill that contains guidelines on how to write/structure JuMP code.
Quite frankly, in my experience, agents tend to be relatively OK at fixing errors when they trigger an error. Errors related to mathematical validity or slow performance (which don’t cause runtime errors) are a lot harder to detect.
Do you use custom skills?
I myself have not taken the time to write one, but it is something I’ve been meaning to do. I’ve seen quite a few skills in the cuOpt source repository (not JuMP-specific ones).
Do you have suggestions for how we could improve AI integration with JuMP?
I believe a few publicly-available skills would go a long way in improving day-to-day workflows and reducing the 80% most common errors.
A longer-term effort (likely to involve multiple people) would be to build a dataset/benchmark to evaluate agent performance on some JuMP or optimization tasks. I know there’s been quite a bit of work on this (especially around the modeling aspect), haven’t kept up with the literature recently.
All along, modeling languages have been striving to adapt to the mature usage paradigm of users for OR/optimization problems. The technological revolution triggered by AI large models is giving rise to a new paradigm for OR/optimization, but it seems that it has not yet converged into a stable form (is that so? I don’t know). Personally, I think that OR/optimization and JuMP/MathOptInterface, as the link between theory and application, should focus both these two aspects. In the theoretical aspect, perhaps they should learn from how AI helps mathematics, and can better assist AI in the mathematical analysis, simplification derivation, and equivalent transformation of the models? In the application aspect, perhaps it is necessary to help AI reduce incorrect modeling of real cases, and for certain mathematical optimization problems, numerical solvers may make mistakes (these examples do exist), LLM and humans often have difficulty distinguishing such situations, so diagnostics/validity/interpretation is necessary for AI (as @mtanneau mentioned).
This is just my random thoughts, for reference only. Finally, I would like to express my gratitude to the developers of JuMP for their hard work!
In addition Kaimon.jl is a promising app that opens the gate between AI agents and Julia. Shall we achieve more exciting functions if JuMP is integrated in it through an extension?
).
!