Tell Julia to use more memory than physical RAM, swapping


#1

Hello.

In R there is a setting, utils::memory.limit(20000), to let R use a quantity of memory larger than the RAM, if needed.
The swap file on the SSD is much slower but sometimes is the only option if you want to perform a complex calculation, for example fitting a regression model with a large dataset and many variables.
For example now I need to use the package MixedModels to fit a large dataset and I can only work with around 100000 rows, but I have 2 million. The same goes for complex bayesian or survival analysis packages.

Is there any option like that on Julia with Windows?


#2

I don’t know about the memory limit but perhaps you would be interested in OnlineStats which is used to analyze datasets larger than memory.


#3

Yes, I asked about that some weeks ago but for now is not able to work with packages such as MixedModels. They told me only linear operations have been implemented.


#4

Windows should (as all modern operating systems) handle this situation automatically i.e.partially/fully swapping out of ram program memory. By default, R uses only RAM (afaik), hence the option. However, Windows is notorious for its poor performance once the limits of RAM are exceeded.

The cheapest option over long term is to add more RAM :wink: Other options are (if possible), using a server with lots of ram or tweaking data types to reduce memory to the extreme i.e. lower precision float’s (i.e. Float16), unsinged integers etc.