The time is spent in (a) deciding which constraints to construct (b) building the model and (c) writing it to a file.
Assuming that (b) only consists of calls to JuMP macros and (c) is not easily modifiable, the major speedup I’m looking for is in (a).
It boils down to what I’ve condensed in my post: based on a condition a constraint is constructed in a specific way. I have (for a simplified model) around 1e9 constraints. They are constructed based on inputs from various configuration files.
Assuming that the way of storing/accessing these needed input data is as fast as possible (I’m converting some code from C++ here, so I’m not fully set on the correct representation - it’s a collection of mostly sparse and a few dense 2d matrices) the bottleneck is checking each set of inputs for “condition” and deciding how the constraint should look like. This is extremely inefficient (no matter the data access) when done in a non parallel way, since I have access to at least 256 cores.
The problem is how to represent the final constraints in a way that I can (after deciding in separate processes) efficiently just pass to jump to create them.
I can of course create the full matrix of coefficients and try to work directly with MOI, but I’m not sure that’s a good idea.
(I can of course come up with a simple in-code example tomorrow, I’m just unfortunately not at my PC anymore)