I am currently using Julia to download, extract and resort data. The procedure is as follows:
For each day
a) Download data for every hour
b) Extract hourly data and combine into a single matrix
c) From that single matrix, extract data for specific regions
d) Save data into NetCDF files
Rinse and repeat for each day
The kicker is that each hourly matrix is 1440x720, so the total size of the daily array is 1440x720x24 which is around 50-100 MB at most depending on the format I use. The problem is that after several loops, the job run crashes due to a lack of memory.
I have to note that it’s not Julia that is crashing, but the server itself. Julia does not return an error - the server simply force-quits the program. But I have no idea why.
Does anyone have any ideas as to why this could happen? Julia works just fine on my personal computer, but I’d rather not download 2-3 years worth of data on my computer and leave it overnight.
Increasing swap memory might keep it from crashing, but it might also run slow if the system start thrashing. You should probably do that anyway, then look at any way you can reuse memory, basically try to reduce the “allocs” and “frees” that are inside loops…
Under linux it is usually a Swap Partition, but you can use a swap file which is nice because you don’t need to resize your disk. Under Windows 7 it’s…fricken hidden…start at the System Properties “dialog”, select the Advanced tab, Click Settings in the Performance section, select Advanced in the new dialog, and it’s Virtual Memory.
In general any memory allocations that are not on the heap, which with Julia usually means mutable structures and arrays. So if you are creating temporary arrays or temporary mutable objects inside a loop, and not referencing them once you leave the loop, they need to be allocated in memory, then outside of the loop are freed. And depending on how the garbage collection runs might not be freed for a while. So if you go back into the loop, they will all be allocated again but the old version will still be hanging around taking up memory…until memory is filled. So if you are using these temporary objects it’s best to allocate them once outside of the loop and reuse them while inside the loop.
Is it the same version of Julia and the packages on both computers?
I am assuming the server has more RAM than your PC and it is not limited per user/process to a smaller number. You may want to check these things, eg on a Linux server with ulimit and cgroups, or ask the server admins.
Other than this, you could track free memory when debugging, eg