Question: JuliaDB and regression models



How do you use JuliaDB for tasks such as fitting a regression model with random effects? (like MixedEffects.jl but with datasets larger than memory)

Disk based data manipulation framework needed

I believe still the MixedModels etc is only compatible with DataFrames, but that’s changing with the Tables.jl / StatsModels changes.


Could add more information, please?
Does it mean we will be soon able to use MixedModels in a different way or that JuliaDB/OnlineStats will be able to use MixedModels?
How can we do it?


IIUC it will be fit(LinearMixedModel, @formula(Y ~ 1 + (1|G)), table) where table can be any table that abides to the Tables.jl format, i.e. a DataFrame or a JuliaDB table or whatever.


Do you have any full example?
It takes the input data from Tables and JuliaDB… but what about the fitting algorithm? Can it work with larger than memory datasets or does it need that the working matrices fit on memory?
We need something that performs every operation on disk or by chunks.
I’ve never tried Tables.jl but I’ll try.
What is IIUC?