On this question, I really appreciate the design of Roots.jl. Its high-level API is opinionated and throws on failure, while its lower-level API gives advanced users more control over how to handle non-convergence.
I think that is a very good pattern. A high-level function should usually be safe and explicit by default. But lower-level interfaces can expose alternative failure-handling strategies for users who really need them.
If you do choose to throw, I would also recommend using a custom exception type. That way, downstream code can still recover cleanly with try/catch when appropriate, without having to inspect error strings or catch unrelated exceptions.
So for me, it depends on the API’s abstraction level.
Similar to Roots.jl, I also prefer for internal functions to maybe also just return the error or in some way information what/why things failed – and the high level parts to throw those.
I think I will return the tuple <result>, error. I cannot use NaN because my result is not a number. Then the caller can decide to print a warning in case of an error, or to retry with different parameters.