You should generally use qr(X, Val(true)) \ y to get a least-square solution (defaulting to the minimum-norm solution if it is not unique, IIRC). For non-square X, it is equivalent to simply do X \ y.
You almost certainly do not want to form X'X at all if it is nearly singular (i.e. if X is badly conditioned), although this is deceptively the way least-squares problems are usually taught. See also this post.