As pointed out by @antoine-levitt, what you are proposing is a kind of gradient descent, except with very small stepsizes. Compared to conventional (quasi-Newton, trust-region) methods, you are losing efficiency on two fronts:
-
gradient descent has significantly worse (local) convergence properties,
-
a lot of unnesessary evaluations are done for interim points you do not really care about.
Also, the lackluster performance for a simple linear system should be a warning: it only gets worse for difficult nonlinear problems.