Solving nonconvex quadratic programs is difficult. Gurobi takes longer because it can find the global optimum.
So you can choose between fast local optimum or slower global optimum.
There might be other algorithms in NLopt that find better solutions. You should try a few out