Convex.jl and Icnn

Specific Domains Optimization (Mathematical)
1

Dear all,

I would like to implement an Input Convex Neural Networks ICNN with the awesome convex.jl tool. The purpose to minimize a cost function. However, in order to get convexity with Icnn, weighting matrices are nonnegative and activation function is convex and nonnegative. So within convex, there is the pos(x) and sound like relu activation function.

I did the following evaluation:

using Flux
using Convex

W = NNlib.relu.(rand(3,3))
b = rand(3,1)

x = Variable(3,1)
y = Variable(3,1)

expr = Convex.pos(W * x[:,1] + b)

Then I tried :

y[:,1] == Convex.pos(W * x[:,1] + b)

And I’ve got the warning

┌ Warning: Expression not DCP compliant. Trying to solve non-DCP compliant problems can lead to unexpected behavior.

I’m not a professional with DCP subject. However I do not understand why added the convex.jl constraint lead to “warming” as ‘y’ is a variable. Maybe a clever person can help me.

Warmest regards

2

Hi!
I don’t know anything about ICNNs but I think I see where your problem is.
When doing convex optimization, you are allowed two types of constraints: affine equality constraints and convex inequality constraints. This ensures that the feasible region you define is a convex set.
The function f(x) -> pos(Wx+b) is convex but not affine, so I guess that is why the equality constraint y == f(x) is not accepted by Convex.jl? Could you try with an inequality just to check?

3

Thanks for answering.

Please find the inequality:

constraint = Constraint[ y[:,1] >= expr   ]
1-element Vector{Constraint}:
 >= constraint (convex)
├─ index (affine; real)
│  └─ 3-element real variable (id: 175…826)
└─ max (convex; positive)
   ├─ + (affine; real)
   │  ├─ * (affine; real)
   │  │  ├─ …
   │  │  └─ …
   │  └─ [0.34773; 0.713025; 0.600791;;]
   └─ 0

and

julia> constraint = Constraint[ y[:,1] <= expr   ]
1-element Vector{Constraint}:
┌ Warning: Expression not DCP compliant. Trying to solve non-DCP compliant problems can lead to unexpected behavior.

So <= provides a DCP compliant issue.

I also evaluate:

julia> constraint = Constraint[  expr >= ones(3,1)]
1-element Vector{Constraint}:
┌ Warning: Expression not DCP compliant. Trying to solve non-DCP compliant problems can lead to unexpected behavior.

and

constraint = Constraint[  expr <= ones(3,1)]
1-element Vector{Constraint}:
 <= constraint (convex)
├─ max (convex; positive)
│  ├─ + (affine; real)
│  │  ├─ * (affine; real)
│  │  │  ├─ …
│  │  │  └─ …
│  │  └─ [0.34773; 0.713025; 0.600791;;]
│  └─ 0
└─ [1.0; 1.0; 1.0;;]
4

Yeah, just as @gdalle said, expressions of the form convex_function_of_variables >= 0 are not allowed (and if zero isn’t on one side, we subtract terms over to the other so it is), because that region is not a convex set. Think about a disk: the equation for (x,y) to be in a circular disk of radius r is x^2 + y^2 <= r^2:

Screen Shot 2022-05-12 at 15.56.31
Screen Shot 2022-05-12 at 15.56.31618×502 3.13 KB

For points to be outside the disk, x^2 + y^2 > r^2:
Screen Shot 2022-05-12 at 15.56.33

Clearly this shape is not convex.

See The DCP ruleset — CVX Users' Guide for all the DCP rules.

2 Likes
5

Yeah, sorry I should have been clearer: you are allowed constraints of the form f(x) <= 0 where f is convex. Not f(x) >= 0

1 Like