How to solve Ax=b, with A,x,b are matrix

Please check PSA: make it easier to help you
In particular, format code using triple backticks to ease readability.
Further, the point that mentions how to write a minimal reproducible example. Your example is too complex for the problem you are asking help with.

I can only see that you did a\b in the fifth line below function grad...
That line will run correctly if you do (permutedims(yourgradient)\yourhessian). This is because of the shape of the arrays you are trying to use.

But how did you end up writing all of that code before managing to run the first 5 lines inside your function? Did it work before and you did some change?

2 Likes