It’s a known issue that StatsModels has high compilation costs currently:
https://github.com/JuliaStats/StatsModels.jl/issues/201
And I’m afraid it’s not been optimized at all for such a large number of variables.
Maybe you’d better compute matrix columns manually, possibly based on the matrix that StatsModels generates from an elementary pattern.