@miles.lubin I’m probably missing something, but I’m not seeing how your suggestion to re-express the problem as min ||y||^2 s.t. y == Ax-b is helpful for memory management. Is it because I’m not matrix multiplying the constraints? (I thought this wasn’t allowed)
If I run something akin to the file I previously shared, JuMP takes forever if n is sufficiently large, because there are n constraints (dataXparms[i=1:size(X,1)]==...) on the objective function, where n=size(X,1). Moreover, JuMP appears to use more memory in this instance (presumably because of the large number of constraints). I’m not sure if the regression applications you’ve used have relatively small n.
For n=300: JuMP uses 70.3 MB and takes 3.6s when the additional n constraints are used.
For n=3000: JuMP’s memory usage approached 2 GB and was still running after 300s when I gave up.
If I put y[i]-sum(X[i,k]*ß[k] for k=1:size(X,2)) directly into the objective function, then the usage is:
n=300: 8 MB, time is <1s
n=3000: 70 MB, time is 1.2s
n=30000: 654 MB, time is 10.8s
Related to OP, the JuMP memory usage is ~30x-40x larger than the data itself. Adding the n constraints doesn’t seem to solve the problem. It appears to be directly proportional to n, since a 10x scale-up of n roughly increases JuMP’s usage by a factor of 10.
I’m not sure that this memory usage is a problem per se, but I don’t recall encountering this in previous version of JuMP.
Big shout out to you and Iain and Joey et al. for developing this—it’s amazing!