I use mixed models on a large file (500000 rows).
My model formula looks like this:
Y ~ 0 + num1:factor1 + num1:factor2 + num2:factor3 + factor4 + (0 + num3|subject) + (0 + num4|subject) + (1|subject),
num - numeric variables;
factor - categorical variables/factors.
Since categorical variables have many unique levels, the fixed effects matrix is very sparse (sparsity ~0.9).
Fitting such a matrix if it is handle as dense requires a lot of time and RAM.
I had the same problem with linear regression.
My dense matrix was
20GB, but when I converted it to sparse it became only
So, I implemented regression in
R using following functions:
sparse.model.matrix(to create a sparse model/design matrix) and
MatrixModels:::lm.fit.sparse(to fit a sparse matrix and calculate coefficients).
Can I apply a similar approach to mixed models and realised it using Julia packages?
What functions / packages can I use to implement this?
That is, my main question is whether it is possible to implement mixed models with sparse matrices?
What package/functions should I use to create X and Z sparse model matrices?
Then, which function should I use for fitting the model with sparse matrices to get coefficients?
I would be very-very grateful for any help with this!