# OLS: difference between \\ and normal equation (and inv or pinv)
**URL:** https://discourse.julialang.org/t/ols-difference-between-and-normal-equation-and-inv-or-pinv/66739
**Category:** General Usage
**Tags:** linearalgebra
**Created:** [August 20, 2021, 3:51am UTC](https://discourse.julialang.org/t/ols-difference-between-and-normal-equation-and-inv-or-pinv/66739 "2021-08-20T03:51:11Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@Alec\_Loudenback](https://discourse.julialang.org/u/Alec_Loudenback)
#### Post date: [August 20, 2021, 3:51am UTC](https://discourse.julialang.org/t/ols-difference-between-and-normal-equation-and-inv-or-pinv/66739/1 "2021-08-20T03:51:11Z")
> [@Claim (false): Julia isn't multiple dispatch but overloading](https://discourse.julialang.org/t/claim-false-julia-isnt-multiple-dispatch-but-overloading/42370/114):
>
> `ols(x,y) = inv(x'x)x'y`
Hello, I am hoping to reference this in an [upcoming article on JuliaActuary](https://juliaactuary.org/blog/julia-getting-started-actuaries/#multiple_dispatch). Is there something fundamentally different in the above code vs `ols(x,y) = x \ y` that would make this syntactically shorter version able to be used but without distorting the message?
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [August 20, 2021, 3:55am UTC](https://discourse.julialang.org/t/ols-difference-between-and-normal-equation-and-inv-or-pinv/66739/2 "2021-08-20T03:55:44Z")
It seems to me `ols(x,y) = inv(x'x)x'y` is a bit questionable example to use. I don’t think compilers are smart enough to figure out that this really should be implemented as `ols(x,y) = (x'x)\x'y`, or even better with something like QR without forming the normal matrix. It looks neat, but it really isn’t.
---
### Author:  [@Alec\_Loudenback](https://discourse.julialang.org/u/Alec_Loudenback)
#### Post date: [August 20, 2021, 4:10am UTC](https://discourse.julialang.org/t/ols-difference-between-and-normal-equation-and-inv-or-pinv/66739/3 "2021-08-20T04:10:31Z")
is there another succinct example that you think captures the same ideas but doesn’t have the issues that you point out?
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [August 20, 2021, 4:31am UTC](https://discourse.julialang.org/t/ols-difference-between-and-normal-equation-and-inv-or-pinv/66739/4 "2021-08-20T04:31:59Z")
I think this is neat:
```julia
n = 13;
x = rand(n); # independent variable
y = 6 * x + 0.71 * (rand(n) .- 0.5)/2; # fake data: note the slope of 6, random noise
A = [x ones(n)]; # coeff matrix of system of too many equations for linear fit
A \ y # solve for the fit parameters using least squares; note the slope ~ 6
2-element Vector{Float64}:
5.8523271815862445
0.06492552228958647
```
🙂 Julia is clever.
---
### Author:  [@apo383](https://discourse.julialang.org/u/apo383)
#### Post date: [August 20, 2021, 5:46am UTC](https://discourse.julialang.org/t/ols-difference-between-and-normal-equation-and-inv-or-pinv/66739/5 "2021-08-20T05:46:33Z")
> [@PetrKryslUCSD](#):
>
> smart enough to figure out… `ols(x,y) = (x'x)\x'y`
I wonder if that could be figured out though? The adjoint is already lazy, `\` has methods for QR or LU factorizations, and there is [`LazyArrays.jl`](https://github.com/JuliaArrays/LazyArrays.jl). Maybe it’s not so far-fetched to make `A'*` and `inv` lazy, so that the pseudo-inverse is automatically recognized and does the right thing. It seems like `inv(A'A)*`, and maybe the whole pattern `(x'x)\x'` is lazy-able, as well as right pseudo-inverse. That laziness might do operations in place, so you don’t have to muck up the beautiful `*` syntax with `mul!`. (There’s probably already an old thread on this in discourse, but I couldn’t find it.)
In teaching, I’ve taken to avoid writing A^{-1}b on the whiteboard, because when students implement it in code they always do `inv`. So I just write `\` as if it’s math (after checking there are no real mathematicians around). I think Matlab’s introduction of `\` and overloading it with pseudo-inverse was a true stroke of genius. But assuming the world isn’t going to adopt `\`, maybe Julia can cleverly do the right thing with `inv(A)*b` and pseudo-inverses.
---
### Author:  [@Alec\_Loudenback](https://discourse.julialang.org/u/Alec_Loudenback)
#### Post date: [August 20, 2021, 12:35pm UTC](https://discourse.julialang.org/t/ols-difference-between-and-normal-equation-and-inv-or-pinv/66739/6 "2021-08-20T12:35:25Z")
So this one has the advantage of being smarter about the operations due to multiple dispatch than the original? Is that because the two (potentially different) types are called with a single method instead of intermediate methods like `inv`, which operates only on the first argument/type first?
---
### Author:  [@bgroenks](https://discourse.julialang.org/u/bgroenks)
#### Post date: [August 20, 2021, 1:37pm UTC](https://discourse.julialang.org/t/ols-difference-between-and-normal-equation-and-inv-or-pinv/66739/7 "2021-08-20T13:37:26Z")
The intention was not to provide a fully optimized form of least squares, which would indeed look a lot more involved. Ideally, you should first compute the LU or QR decomposition, for example, followed by `ldiv!`, and everything should also be done in-place.
The point was just to provide a simple example where you get lots of specialization for free. This is in fact the case with `ols(x,y) = inv(x'x)x'y`, as can be seen by simply observing the dozens of specializations provided by `LinearAlgebra` for `inv`, `adjoint`, and `*`.
To answer @Alec_Loudenback’s original question, I see no reason why you couldn’t use the shorter form `\` and make the same point. I used the slightly longer form because it involved more than one operation, which I thought might be more illustrative.
---
### Author:  [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)
#### Post date: [August 20, 2021, 1:49pm UTC](https://discourse.julialang.org/t/ols-difference-between-and-normal-equation-and-inv-or-pinv/66739/8 "2021-08-20T13:49:01Z")
I get it. I just cringe when I see `inv` in implementation (not in pseudocode).
So how about `ols(x, y) = pinv([x ones(length(x))]) * y`?
---
### Author:  [@bgroenks](https://discourse.julialang.org/u/bgroenks)
#### Post date: [August 20, 2021, 2:01pm UTC](https://discourse.julialang.org/t/ols-difference-between-and-normal-equation-and-inv-or-pinv/66739/9 "2021-08-20T14:01:14Z")
Yep, that works too!
I also like the added bonus of not needing to pre-pad `x` with the intercept column 👍.
---
### Author:  [@jlperla](https://discourse.julialang.org/u/jlperla)
#### Post date: [August 20, 2021, 4:27pm UTC](https://discourse.julialang.org/t/ols-difference-between-and-normal-equation-and-inv-or-pinv/66739/10 "2021-08-20T16:27:21Z")
Depending on the goals of your article, it might make sense to instead show a little bit more about factorizations/etc. which are connected to the multiple dispatch in this case. These are at the heart of all of these operations, especially if you think “users” would care about big problems or end up solving least squares on the same matrix with different RHSs.
See [22. Numerical Linear Algebra and Factorizations — Quantitative Economics with Julia](https://julia.quantecon.org/tools_and_techniques/numerical_linear_algebra.html) for some stuff on these factorizations, and [23. Krylov Methods and Matrix Conditioning — Quantitative Economics with Julia](https://julia.quantecon.org/tools_and_techniques/iterative_methods_sparsity.html#Iterative-Methods-for-Linear-Least-Squares)
There are no other languages like julia for making using more “advanced” algorithms accessible. And advanced is in the eye of the beholder. To many people, the idea that you would solve least squares without manually forming the normal euqations is tough to imagine (as is the subtlety of why it is slower and/or could have catastrophic numerical error is another issue entirely). By using good packages and algorithms, people can stay more ignorant of these issues.
But, as I said, it all depends on what your goals are. If this is narrowly talking about multiple dispatch in that situation, then I suggest showing the OLS with a sparse vs. a dense matrix.
---
### Author:  [@Alec\_Loudenback](https://discourse.julialang.org/u/Alec_Loudenback)
#### Post date: [August 22, 2021, 12:06am UTC](https://discourse.julialang.org/t/ols-difference-between-and-normal-equation-and-inv-or-pinv/66739/11 "2021-08-22T00:06:39Z")
Thanks, all. Seems like the original example is a bit suboptimal, not because of multiple dispatch, but because there are better alternatives to calling `inv` for this problem which are more directly performing the optimized maths.