One runs into these kind of problems all the time with complex models in economics — some parameters just don’t have an equilibrium, and it is not practically possible to treat this as a constraint.
In my own packages I am experimenting with allowing objective functions or residuals (for nonlinear solvers) to return nothing in this case — this is a bit more disciplined than 1e20 or NaN. It would be great if there was a standard way of handling this though across various optimization and rootfinding packages.