I’m trying to optimize a function with respect to a real matrix M with all eigenvalues real and lying in the open interval (0,1).

I know that for a real symmetric positive-definite matrix S, we can use the Cholesky factorization S = L L^T where elements of L are free parameters.

Is there a similar way of enforcing the above eigenvalue constraint?

Use semidefinite programming with the following constraints:

where I is the identity matrix and S and T are matrix decision variables.

Thank you for your reply. So to optimize over matrices M which satisfy my eigenvalue constraint, I should equivalently optimize over general positive (semi-) definite matrices T and S, subject to the linear constraint that T=I-S?

Yes. S will be your M.