I am currently using Julia to download, extract and resort data. The procedure is as follows:
- For each day
a) Download data for every hour
b) Extract hourly data and combine into a single matrix
c) From that single matrix, extract data for specific regions
d) Save data into NetCDF files
- Rinse and repeat for each day
The kicker is that each hourly matrix is 1440x720, so the total size of the daily array is 1440x720x24 which is around 50-100 MB at most depending on the format I use. The problem is that after several loops, the job run crashes due to a lack of memory.
I have to note that it’s not Julia that is crashing, but the server itself. Julia does not return an error - the server simply force-quits the program. But I have no idea why.
Does anyone have any ideas as to why this could happen? Julia works just fine on my personal computer, but I’d rather not download 2-3 years worth of data on my computer and leave it overnight.