Any Julia's equivalent to R's packages mcgv or mixed-effects models larger than memory?


#1

Hello.

I’m interested in fitting mixed effects models with large dataset that don’t fit on memory.
R’s lme4 it’s too slow and doesn’t work if the dataset is large (a fraction of your RAM).
speedglm and mgcv are a little bit faster but still have problems.

I’ve decided to move to Julia to try to find a better option.
Mixedmodels.jl is like lme4.

Does Julia have something more like mgcv, faster and able to run Generalized additive mixed models?
Or something able to fit mixed-effects models with datasets of around 50GB (on a computer with 12GB)?
I mean not loading everything on memory, automatically streaming to disk as necessary.

Another option would be Spark or Flink, they work with very large datasets but I think they don’t have any implementation of mixed-effects models.


#2

You may want to check https://github.com/linkedin/photon-ml

I haven’t used it myself but they claim to be able to work with very large data sets.


#3

By the way, although MixedModels.jl is similar in design to lme4 (not surprising given my involvement in both projects) it is much more careful of the storage usage and generally much faster.