Pre-processing time and Mosek Solve inordinately long

Specific Domains Optimization (Mathematical)
11

@blegat Ok, this turns the problem into something that precompiles quickly, but now we have over 3 million constraints.

Problem
  Name                   :
  Objective sense        : minimize
  Type                   : CONIC (conic optimization problem)
  Constraints            : 3209618
  Affine conic cons.     : 0
  Disjunctive cons.      : 0
  Cones                  : 0
  Scalar variables       : 1396
  Matrix variables       : 400 (scalarized: 251185564)
  Integer variables      : 0

I don’t even want to conjecture how long it is going to take Mosek to solve this problem, but I will let you know when it finishes running if it is done by tomorrow morning, else I am going to end it.

One of the main hopes of going to Julia for this problem was to avoid the memory issues we run into with Mosek due to the precompilation from YALMIP when we begin to increase the number of basis elements in the polynomial. So, it kind of feels a bit like we are out of the frying pan and into the fire.

What is so different about the Newton polytope reduction between YALMIP and JuMP to make the presolve time so unmanageably long?

Edit: That code, the zsh shell crashed, not sure why, could just be a fluke, would have to run it again. Instead, I decided to do some optional switching so that 2 of the constraints could have the post_filter on and the rest don’t. This one took a couple hours to pre-process, with the following problem handed to Mosek:

  Name                   :
  Objective sense        : minimize
  Type                   : CONIC (conic optimization problem)
  Constraints            : 3203458
  Affine conic cons.     : 0
  Disjunctive cons.      : 0
  Cones                  : 0
  Scalar variables       : 1398
  Matrix variables       : 396 (scalarized: 251137160)
  Integer variables      : 0

Optimizer started.
Presolve started.
Linear dependency checker started.
Linear dependency checker terminated.
Eliminator started.
Freed constraints in eliminator : 0
Eliminator terminated.
Eliminator - tries                  : 1                 time                   : 0.00
Lin. dep.  - tries                  : 1                 time                   : 0.66
Lin. dep.  - primal attempts        : 1                 successes              : 1
Lin. dep.  - dual attempts          : 0                 successes              : 0
Lin. dep.  - primal deps.           : 0                 dual deps.             : 0
Presolve terminated. Time: 2.26

The shell also killed the program again, i.e. the problem was not solved.

I added a full Profile.print() output log to the Github, for the full problem with just the symmetries, not the certificate (it did finish running last night). Not sure if there is anything else in there to see, I think you were correct most of the overhead is in the Newton polytope reduction.