Memory Error

General Usage Performance
1

Hi!

I am currently using Julia to download, extract and resort data. The procedure is as follows:

  1. For each day
    a) Download data for every hour
    b) Extract hourly data and combine into a single matrix
    c) From that single matrix, extract data for specific regions
    d) Save data into NetCDF files
  2. Rinse and repeat for each day

The kicker is that each hourly matrix is 1440x720, so the total size of the daily array is 1440x720x24 which is around 50-100 MB at most depending on the format I use. The problem is that after several loops, the job run crashes due to a lack of memory.

I have to note that it’s not Julia that is crashing, but the server itself. Julia does not return an error - the server simply force-quits the program. But I have no idea why.

Does anyone have any ideas as to why this could happen? Julia works just fine on my personal computer, but I’d rather not download 2-3 years worth of data on my computer and leave it overnight.

Cheers

2

This seems like a case where preallocation might help

3

How would this be done? I do preallocate the arrays in my module.

Or would this be because of logging issues? I do a lot of logging, but the logs only are printed at the end of each job so that could contribute to memory loss.

4

Increasing swap memory might keep it from crashing, but it might also run slow if the system start thrashing. You should probably do that anyway, then look at any way you can reuse memory, basically try to reduce the “allocs” and “frees” that are inside loops…

5

I don’t quite understand what you mean. What exactly are “allocs” and “frees”?

And what is swap memory?

6

Under linux it is usually a Swap Partition, but you can use a swap file which is nice because you don’t need to resize your disk. Under Windows 7 it’s…fricken hidden…start at the System Properties “dialog”, select the Advanced tab, Click Settings in the Performance section, select Advanced in the new dialog, and it’s Virtual Memory.

In general any memory allocations that are not on the heap, which with Julia usually means mutable structures and arrays. So if you are creating temporary arrays or temporary mutable objects inside a loop, and not referencing them once you leave the loop, they need to be allocated in memory, then outside of the loop are freed. And depending on how the garbage collection runs might not be freed for a while. So if you go back into the loop, they will all be allocated again but the old version will still be hanging around taking up memory…until memory is filled. So if you are using these temporary objects it’s best to allocate them once outside of the loop and reuse them while inside the loop.

7

not on the stack?

8

Is it the same version of Julia and the packages on both computers?

I am assuming the server has more RAM than your PC and it is not limited per user/process to a smaller number. You may want to check these things, eg on a Linux server with ulimit and cgroups, or ask the server admins.

Other than this, you could track free memory when debugging, eg

julia> round(Int, Base.Sys.free_memory() / 2^20)
8027

julia> A = ones(1000, 1000, 1000);

julia> round(Int, Base.Sys.free_memory() / 2^20)
1966

You should then make sure that objects which are no longer used can be garbage collected.

9

As @tamas_papp says you need to look at the limits for your user session on this server.
Could you please tell us more about the server - which OS It is running, how much RAM it has

cat /etc/elease
uname -r
free -g

Can we assume you are starting an ssh session on the server, then running ‘screen’ or something similar?

10

Okay, after going an entire circle, it was found that the memory error is due to NetCDF’s memory leakage ahha. Thank you to everyone for your responses!

1 Like