# Implement Mixed Models with sparse X and Z matrices

**URL:** <https://discourse.julialang.org/t/implement-mixed-models-with-sparse-x-and-z-matrices/49539>\
**Category:** Machine Learning\
**Tags:** question, package, sparse\
**Created:** [November 3, 2020, 4:28pm UTC](https://discourse.julialang.org/t/implement-mixed-models-with-sparse-x-and-z-matrices/49539 "2020-11-03T16:28:24Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Antonina\_Klyuyeva](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antonina_klyuyeva/32/18875_2.png) [@Antonina\_Klyuyeva](https://discourse.julialang.org/u/Antonina_Klyuyeva)\
**Post date:** [November 3, 2020, 4:28pm UTC](https://discourse.julialang.org/t/implement-mixed-models-with-sparse-x-and-z-matrices/49539/1 "2020-11-03T16:28:25Z")

</div>

Hi there,  
I use mixed models on a large file (500000 rows).  
My model formula looks like this:  
`Y ~ 0 + num1:factor1 + num1:factor2 + num2:factor3 + factor4 + (0 + num3|subject) + (0 + num4|subject) + (1|subject)`,  
where `num` - numeric variables; `factor` - categorical variables/factors.

Since categorical variables have many unique levels, the fixed effects matrix is ​​very sparse (sparsity ~0.9).  
Fitting such a matrix if it is handle as dense requires a lot of time and RAM.

I had the same problem with linear regression.  
My dense matrix was `20GB`, but when I converted it to sparse it became only `35MB`.  
So, I implemented regression in `R` using following functions:

1. `sparse.model.matrix` (to create a sparse model/design matrix) and
2. `MatrixModels:::lm.fit.sparse` (to fit a sparse matrix and calculate coefficients).

Can I apply a similar approach to mixed models and realised it using Julia packages?  
What functions / packages can I use to implement this?

That is, my main question is whether **it is possible to implement mixed models with sparse matrices**?  
What package/functions should I use to create X and Z sparse model matrices?  
Then, which function should I use for fitting the model with sparse matrices to get coefficients?

I would be very-very grateful for any help with this!

---

<div class="post-metadata">

**Author:** ![dmbates](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmbates/32/44_2.png) [@dmbates](https://discourse.julialang.org/u/dmbates)\
**Post date:** [November 3, 2020, 5:49pm UTC](https://discourse.julialang.org/t/implement-mixed-models-with-sparse-x-and-z-matrices/49539/2 "2020-11-03T17:49:57Z")

</div>

There is some provision in the [`MixedModels` package](https://github.com/JuliaStats/MixedModels.jl) for working with sparse model matrices for the fixed-effects parameters. I haven’t tried it out myself and am not sure how well it is integrated with the [`StatsModels` package](https://github.com/JuliaStats/StatsModels.jl) which does the conversion from formula/data to model matrices. Perhaps @palday or @dave.f.kleinschmidt may be able to provide more detail on how easy or difficult it would be.

---

<div class="post-metadata">

**Author:** ![Juan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juan/32/7657_2.png) [@Juan](https://discourse.julialang.org/u/Juan)\
**Post date:** [November 3, 2020, 7:07pm UTC](https://discourse.julialang.org/t/implement-mixed-models-with-sparse-x-and-z-matrices/49539/3 "2020-11-03T19:07:43Z")

</div>

You should try with  
[https://github.com/FixedEffects/FixedEffectModels.jl](https://github.com/FixedEffects/FixedEffectModels.jl)

---

<div class="post-metadata">

**Author:** ![Antonina\_Klyuyeva](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antonina_klyuyeva/32/18875_2.png) [@Antonina\_Klyuyeva](https://discourse.julialang.org/u/Antonina_Klyuyeva)\
**Post date:** [November 3, 2020, 7:34pm UTC](https://discourse.julialang.org/t/implement-mixed-models-with-sparse-x-and-z-matrices/49539/4 "2020-11-03T19:34:42Z")

</div>

@Juan Does this package can handle mixed model formulas (I mean, random effects part)?

---

<div class="post-metadata">

**Author:** ![Antonina\_Klyuyeva](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antonina_klyuyeva/32/18875_2.png) [@Antonina\_Klyuyeva](https://discourse.julialang.org/u/Antonina_Klyuyeva)\
**Post date:** [November 3, 2020, 7:39pm UTC](https://discourse.julialang.org/t/implement-mixed-models-with-sparse-x-and-z-matrices/49539/5 "2020-11-03T19:39:06Z")

</div>

@dmbates Thank you for the answer, I’m using the `MixedModels` package now, but it looks like it handles `X` matrix as dense, since model fitting consumes enormous RAM.

---

<div class="post-metadata">

**Author:** ![palday](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palday/32/12640_2.png) [@palday](https://discourse.julialang.org/u/palday)\
**Post date:** [November 3, 2020, 7:40pm UTC](https://discourse.julialang.org/t/implement-mixed-models-with-sparse-x-and-z-matrices/49539/6 "2020-11-03T19:40:31Z")

</div>

FixedEffectModels doesn’t – and note that “fixed effects” in econometrics has a different meaning that elsewhere (it’s “categorical fixed effect” in the terminology in MixedModels).

The Z matrices in MixedModels are already sparse (that’s about half the magic of the MixedModels approach compared – @dmbates developed a way to express fitting as a sparse penalized least squares problem, while most techniques depend on a dense generalized least squares problem). The X matrix can be sparse, but the formula interface won’t generate it. If you call the `LinearMixedModel` constructor directly with a sparse X that you constructed by hand, it will work though. The support for sparse `FeMat` was one of the changes in MixedModels 3.0

---

<div class="post-metadata">

**Author:** ![palday](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palday/32/12640_2.png) [@palday](https://discourse.julialang.org/u/palday)\
**Post date:** [November 3, 2020, 7:42pm UTC](https://discourse.julialang.org/t/implement-mixed-models-with-sparse-x-and-z-matrices/49539/7 "2020-11-03T19:42:24Z")

</div>

Relevant (merged) pull request: [https://github.com/JuliaStats/MixedModels.jl/pull/309](https://github.com/JuliaStats/MixedModels.jl/pull/309)

This functionality is very new and the people using it so far are close collaborators, so we haven’t yet written good documentation on it.

**EDIT:**

One more tip: use `Grouping()` pseudo-contrasts when you have many levels of the grouping variable: `contrasts=Dict(:subject => Grouping())`

---

<div class="post-metadata">

**Author:** ![palday](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palday/32/12640_2.png) [@palday](https://discourse.julialang.org/u/palday)\
**Post date:** [November 3, 2020, 7:48pm UTC](https://discourse.julialang.org/t/implement-mixed-models-with-sparse-x-and-z-matrices/49539/8 "2020-11-03T19:48:53Z")

</div>

I should also emphasize @Antonina_Klyuyeva that if you run into problems but have a good minimal working example then we’re (well, I) happy to help. And a good MWE is great for improving out support and expanding our tests. Note that your MWE can also include real data, if you’re able to share that.