I’m trying to fit mixed-effect models to large dataset.
R’s lme4 is very slow and can’t deal with more than 50000 rows on my computer. The amount of memory seems to increase with the cube of the number of rows.
Julia’s Mixedmodels is approximatelly 5 times faster but has the same limit.
There are other R’s packages that sometimes can deal with much larger datasets. For example mgcv but it only works well if you have few levels of the random variable.
The other package I’ve found is glmmTMB (Generalized linear mixed models with various extensions, including zero-inflation. ), I’m trying but for now I’ve got to fit the model with 400000 rows, though it took 850 seconds. I think it will be able to fit larger problems.
Is there any “Template Model Builder” package for Julia? I guess it would be faster than its R counterpart.