Do you happen to know if there is another way to use vector-valued expressions, if it is not possible to be done in
JuMP?
You can use vector-valued linear or quadratic expressions in the @constraint macro. You just can’t use vector-valued expressions in the @NLxxx macros.
I would split out the inner inv(B(x)) * x term:
using JuMP
model = Model()
@variable(model, x[1:2] >= 0)
A(x) = [1 x; 0 1]
B(x) = [1 x; 0 1]
b = [1, 2]
@variable(model, y[1:2])
@constraint(model, B(x) * y .== x)
@constraint(model, A(x) * y .== b)
Now you have two non-convex quadratic constraints.