I want to use the eigenvalues of the matrix Z in a constraint (e.g their sum or product) as in the following:

using JuMP
using LinearAlgebra
model = Model()
@variable(model, X[1:2, 1:2] >= 0)
@variable(model, c[1:2] >= 0)
@variable(model, Y[1:2, 1:2])
Z = Y * X + X’ * Y - c * c’
vals, vecs = eigen(Z)
@NLconstraint(model, const1, sum(vals[j] for j = 1:2)>=0)
@NLconstraint(model, const2, prod(vals[j] for j = 1:2)>=0)

But the error says that abs2 is not defined for type GenericQuadExpr. What does this mean? Am I not computing the eigenvalues correctly? If yes, how can I get the eigenvalues and use them

Usually if you want to write something like this in an optimization domain-specific language, you have to define Z and vals as variables too, and add a (NL)constraint that forces them to be equal to their expression.
However, with a 2x2 matrix, you may be better off just writing the explicit expression?

What are you trying to do? Do you want Z to be positive semi-definite? You can do something like:

using JuMP
model = Model()
@variable(model, X[1:2, 1:2] >= 0)
@variable(model, c[1:2] >= 0)
@variable(model, Y[1:2, 1:2])
# @constraint(model, Y * X + X' * Y - c * c' >= 0, PSDCone())
Z = Y * X + X' * Y - c * c'
@constraint(model, Z >= 0, PSDCone())

Using eigen in nonlinear constraints isn’t supported.

Are you sure you want to have those two expressions with the eigenvalues directly? This is just a wild guess, but judging by your problem description it looks like a typical Lyapunov-Inequality for a 2x2 matrix Z. For such problems you’d usually want the Z to be negative definite, which means all eigenvalues to be strictly negative. This can be directly expressed in JuMP. Your NLconstraints look like the result you’d get after applying something akin to Hurwitz Criterion on the characteristic polynomial of Z, which should then yield constraints on the entries of Z, that is why I am not sure you really want to use the eigenvalues directly here.

However, if that is truly the case here, you might want to consider Tr(Z) for the first constraint and det(Z) for the last one, although I doubt both will work. Maybe also try your luck with Convex.jl