Unfortunately, you cannot do this very easily.
You can register a function which returns a scalar: Nonlinear Modeling · JuMP
There is also a hack for user-defined functions which return a vector:
Longer-term, we’re in the middle of a rewrite to JuMP’s nonlinear interface, https://github.com/jump-dev/JuMP.jl/pull/3106, which will make this work with no hacks or registered functions.
(p.s. I was having a conversation with @ccoffrin a few hours ago that this is the most common question I get about JuMP. It comes up a few times every week… So hopefully we’ll have a better answer once that PR is merged.)