# GLM is slow on large datasets. Using OnlineStats for regressions? MixedModels?

**URL:** <https://discourse.julialang.org/t/glm-is-slow-on-large-datasets-using-onlinestats-for-regressions-mixedmodels/17695>\
**Category:** Performance\
**Tags:** glm\
**Created:** [November 19, 2018, 12:49am UTC](https://discourse.julialang.org/t/glm-is-slow-on-large-datasets-using-onlinestats-for-regressions-mixedmodels/17695 "2018-11-19T00:49:40Z")\
**Posts on this page:** 6\
**Page:** 2

<div class="post-metadata">

**Author:** ![andreasnoack](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andreasnoack/32/27_2.png) [@andreasnoack](https://discourse.julialang.org/u/andreasnoack)\
**Post date:** [November 26, 2018, 11:56am UTC](https://discourse.julialang.org/t/glm-is-slow-on-large-datasets-using-onlinestats-for-regressions-mixedmodels/17695/21 "2018-11-26T11:56:45Z")

</div>

> I’m not a numeric analyst, but won’t the condition number of (X’ \* X) \ (X’ \* Y) be the square of that of X \ Y?

It is but it’s usually not a problem in statistical applications. If `X` is almost rank deficient then you’ll have a lot of uncertainty about the parameter estimates. The amplified noise from the squared condition number will be nothing compared to the statistical uncertainty. In some applications, you expect a perfect fit and in that case it can matter if you use QR or the Cholesky.

---

<div class="post-metadata">

**Author:** ![Yifan\_Liu](https://avatars.discourse-cdn.com/v4/letter/y/4da419/32.png) [@Yifan\_Liu](https://discourse.julialang.org/u/Yifan_Liu)\
**Post date:** [November 26, 2018, 4:07pm UTC](https://discourse.julialang.org/t/glm-is-slow-on-large-datasets-using-onlinestats-for-regressions-mixedmodels/17695/22 "2018-11-26T16:07:04Z")

</div>

> **[matrix\_OLS\_NYU\_notes.pdf](https://web.stanford.edu/~mrosenfe/soc_meth_proj3/matrix_OLS_NYU_notes.pdf)**
>
> 151.97 KB

Econometrics 101 is a must-have!

---

<div class="post-metadata">

**Author:** ![BLI](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bli/32/37206_2.png) [@BLI](https://discourse.julialang.org/u/BLI)\
**Post date:** [November 26, 2018, 4:28pm UTC](https://discourse.julialang.org/t/glm-is-slow-on-large-datasets-using-onlinestats-for-regressions-mixedmodels/17695/23 "2018-11-26T16:28:08Z")

</div>

Thanks for tip.

Question: in some algorithms (“chemometrics”, etc.), it is common to use PCR (or PLS). PCR essentially chooses the number of latent variables based on the condition number. SVD is convenient because one can play around with \sigma\_{i-1}/\sigma\_i and find a suitable cut. With a “rank revealing” QR factorization where the diagonal elements of R are sorted in descending order, a similar idea could be used. But the standard QR factorization doesn’t contain a sorting of the diagonal elements of R, right?

---

<div class="post-metadata">

**Author:** ![BLI](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bli/32/37206_2.png) [@BLI](https://discourse.julialang.org/u/BLI)\
**Post date:** [November 26, 2018, 4:30pm UTC](https://discourse.julialang.org/t/glm-is-slow-on-large-datasets-using-onlinestats-for-regressions-mixedmodels/17695/24 "2018-11-26T16:30:22Z")

</div>

That is, of course, a valid point 🙂 .

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [November 26, 2018, 4:42pm UTC](https://discourse.julialang.org/t/glm-is-slow-on-large-datasets-using-onlinestats-for-regressions-mixedmodels/17695/25 "2018-11-26T16:42:32Z")

</div>

This article shows you the equations but doesn’t discuss about the proper numerical methods (such as SVD, QR, Cholesky or sparse matrix methods…) to solve them.

---

<div class="post-metadata">

**Author:** ![Nosferican](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nosferican/32/9275_2.png) [@Nosferican](https://discourse.julialang.org/u/Nosferican)\
**Post date:** [November 26, 2018, 5:54pm UTC](https://discourse.julialang.org/t/glm-is-slow-on-large-datasets-using-onlinestats-for-regressions-mixedmodels/17695/26 "2018-11-26T17:54:17Z")

</div>

That is correct. You can use QR factorization for this,

```julia
invperm(F.jpvt)

```

Gives you the sorted column ranking.

[Previous page](https://discourse.julialang.org/t/glm-is-slow-on-large-datasets-using-onlinestats-for-regressions-mixedmodels/17695.md?page=1)
