Hello! MixedModels.jl much faster, and consume less memory. If Mixed Models.jl applicable for your task it will be better to use MixedModels.jl If you need Satterthwaite ddf approximation, analysis with repeated measurements, or application of some nontrivial covariance structures - Metida.jl solve these problems. I didn’t found MixedEffects.jl maybe you mean MixedModels.jl, anyway if I missed this package, please guide me to it.
Hi! I think this is not ‘hard-coded limitations’. When the model is constructed full Z matrix is used for this. This means that if you have 200k observations and 50k subjects - Z matrix will be 200x50k (10 000 mln numbers) and this matrix will be out of memory. I think this problem can be solved in some cases. So in StatsModels there is no easy way to get a sparse model matrix for categorical terms with many levels.
Ok, so there isn’t any hard coded limits… I think how it is written now isn’t too clear… ir really seems like the one stated are fixed problem limite, like in some commercial software… .
You could write instead something like “a standard pc with xxx GB of RAM can efficiently process problems up to …”
I think I should add some clarification in docs. Also in some cases, it is very difficult to predict how many observations can be handled, because, for example, if the model includes two random factors by 100k levels it will cost twice memory as one. So it is not only a data-dependent problem. The solution is to make a sparse Z matrix, for this own implementation of StatsModels methods should be done - and this is part of work on future.
@sylvaticus If you don’t need structured R-side covariance, then MixedModels.jl already uses optimized sparse methods for Z matrices. We don’t have support for many restricted G-side (random effects) covariance structures out of the box but they’re not too hard to implement. The “DIAG” case here corresponds to zerocorr in MixedModel; the “SI” case here could be implemented without too much difficulty. One of my TODOs is to document how to create new types for restricted G-side covariance.