# Convex.jl and Icnn

Dear all,

I would like to implement an Input Convex Neural Networks ICNN with the awesome convex.jl tool. The purpose to minimize a cost function. However, in order to get convexity with Icnn, weighting matrices are nonnegative and activation function is convex and nonnegative. So within convex, there is the `pos(x)` and sound like relu activation function.

I did the following evaluation:

``````using Flux
using Convex

W = NNlib.relu.(rand(3,3))
b = rand(3,1)

x = Variable(3,1)
y = Variable(3,1)

expr = Convex.pos(W * x[:,1] + b)
``````

Then I tried :

``````y[:,1] == Convex.pos(W * x[:,1] + b)
``````

And I’ve got the warning

``````┌ Warning: Expression not DCP compliant. Trying to solve non-DCP compliant problems can lead to unexpected behavior.
``````

I’m not a professional with DCP subject. However I do not understand why added the convex.jl constraint lead to “warming” as ‘y’ is a variable. Maybe a clever person can help me.

Warmest regards

Hi!
I don’t know anything about ICNNs but I think I see where your problem is.
When doing convex optimization, you are allowed two types of constraints: affine equality constraints and convex inequality constraints. This ensures that the feasible region you define is a convex set.
The function `f(x) -> pos(Wx+b)` is convex but not affine, so I guess that is why the equality constraint `y == f(x)` is not accepted by Convex.jl? Could you try with an inequality just to check?

``````constraint = Constraint[ y[:,1] >= expr   ]
1-element Vector{Constraint}:
>= constraint (convex)
├─ index (affine; real)
│  └─ 3-element real variable (id: 175…826)
└─ max (convex; positive)
├─ + (affine; real)
│  ├─ * (affine; real)
│  │  ├─ …
│  │  └─ …
│  └─ [0.34773; 0.713025; 0.600791;;]
└─ 0

``````

and

``````julia> constraint = Constraint[ y[:,1] <= expr   ]
1-element Vector{Constraint}:
┌ Warning: Expression not DCP compliant. Trying to solve non-DCP compliant problems can lead to unexpected behavior.

``````

So <= provides a DCP compliant issue.

I also evaluate:

``````julia> constraint = Constraint[  expr >= ones(3,1)]
1-element Vector{Constraint}:
┌ Warning: Expression not DCP compliant. Trying to solve non-DCP compliant problems can lead to unexpected behavior.
``````

and

``````constraint = Constraint[  expr <= ones(3,1)]
1-element Vector{Constraint}:
<= constraint (convex)
├─ max (convex; positive)
│  ├─ + (affine; real)
│  │  ├─ * (affine; real)
│  │  │  ├─ …
│  │  │  └─ …
│  │  └─ [0.34773; 0.713025; 0.600791;;]
│  └─ 0
└─ [1.0; 1.0; 1.0;;]

``````

Yeah, just as @gdalle said, expressions of the form `convex_function_of_variables >= 0` are not allowed (and if zero isn’t on one side, we subtract terms over to the other so it is), because that region is not a convex set. Think about a disk: the equation for `(x,y)` to be in a circular disk of radius `r` is `x^2 + y^2 <= r^2`:

For points to be outside the disk, `x^2 + y^2 > r^2`: Clearly this shape is not convex.

See The DCP ruleset — CVX Users' Guide for all the DCP rules.

2 Likes

Yeah, sorry I should have been clearer: you are allowed constraints of the form `f(x) <= 0` where `f` is convex. Not `f(x) >= 0`

1 Like