# Memory Error

**URL:** <https://discourse.julialang.org/t/memory-error/30251>\
**Category:** Performance\
**Created:** [October 24, 2019, 12:00am UTC](https://discourse.julialang.org/t/memory-error/30251 "2019-10-24T00:00:16Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![natgeo-wong](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/natgeo-wong/32/10965_2.png) [@natgeo-wong](https://discourse.julialang.org/u/natgeo-wong)\
**Post date:** [October 24, 2019, 12:00am UTC](https://discourse.julialang.org/t/memory-error/30251/1 "2019-10-24T00:00:16Z")

</div>

Hi!

I am currently using Julia to download, extract and resort data. The procedure is as follows:

1. For each day  
a) Download data for every hour  
b) Extract hourly data and combine into a single matrix  
c) From that single matrix, extract data for specific regions  
d) Save data into NetCDF files
2. Rinse and repeat for each day

The kicker is that each hourly matrix is 1440x720, so the total size of the daily array is 1440x720x24 which is around 50-100 MB at most depending on the format I use. The problem is that after several loops, the job run crashes due to a lack of memory.

I have to note that it’s not Julia that is crashing, but the server itself. Julia does not return an error - the server simply force-quits the program. But I have no idea why.

Does anyone have any ideas as to why this could happen? Julia works just fine on my personal computer, but I’d rather not download 2-3 years worth of data on my computer and leave it overnight.

Cheers

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [October 24, 2019, 12:30am UTC](https://discourse.julialang.org/t/memory-error/30251/2 "2019-10-24T00:30:27Z")

</div>

This seems like a case where preallocation might help

---

<div class="post-metadata">

**Author:** ![natgeo-wong](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/natgeo-wong/32/10965_2.png) [@natgeo-wong](https://discourse.julialang.org/u/natgeo-wong)\
**Post date:** [October 24, 2019, 12:32am UTC](https://discourse.julialang.org/t/memory-error/30251/3 "2019-10-24T00:32:08Z")

</div>

How would this be done? I do preallocate the arrays in my module.

Or would this be because of logging issues? I do a lot of logging, but the logs only are printed at the end of each job so that could contribute to memory loss.

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [October 24, 2019, 12:34am UTC](https://discourse.julialang.org/t/memory-error/30251/4 "2019-10-24T00:34:12Z")

</div>

Increasing swap memory might keep it from crashing, but it might also run slow if the system start thrashing. You should probably do that anyway, then look at any way you can reuse memory, basically try to reduce the “allocs” and “frees” that are inside loops…

---

<div class="post-metadata">

**Author:** ![natgeo-wong](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/natgeo-wong/32/10965_2.png) [@natgeo-wong](https://discourse.julialang.org/u/natgeo-wong)\
**Post date:** [October 24, 2019, 1:02am UTC](https://discourse.julialang.org/t/memory-error/30251/5 "2019-10-24T01:02:55Z")

</div>

I don’t quite understand what you mean. What exactly are “allocs” and “frees”?

And what is swap memory?

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [October 24, 2019, 2:22am UTC](https://discourse.julialang.org/t/memory-error/30251/6 "2019-10-24T02:22:14Z")

</div>

> [@natgeo-wong](#):
>
> And what is swap memory?

Under linux it is usually a Swap Partition, but you can use a swap file which is nice because you don’t need to resize your disk. Under Windows 7 it’s…fricken hidden…start at the System Properties “dialog”, select the Advanced tab, Click Settings in the Performance section, select Advanced in the new dialog, and it’s Virtual Memory.

> [@natgeo-wong](#):
>
> What exactly are “allocs” and “frees”?

In general any memory allocations that are not on the heap, which with Julia usually means mutable structures and arrays. So if you are creating temporary arrays or temporary mutable objects inside a loop, and not referencing them once you leave the loop, they need to be allocated in memory, then outside of the loop are freed. And depending on how the garbage collection runs might not be freed for a while. So if you go back into the loop, they will all be allocated again but the old version will still be hanging around taking up memory…until memory is filled. So if you are using these temporary objects it’s best to allocate them once outside of the loop and reuse them while inside the loop.

---

<div class="post-metadata">

**Author:** ![greg\_plowman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/greg_plowman/32/8100_2.png) [@greg\_plowman](https://discourse.julialang.org/u/greg_plowman)\
**Post date:** [October 24, 2019, 4:25am UTC](https://discourse.julialang.org/t/memory-error/30251/7 "2019-10-24T04:25:44Z")

</div>

> [@pixel27](#):
>
> In general any memory allocations that are not on the heap

not on the stack?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 24, 2019, 5:31am UTC](https://discourse.julialang.org/t/memory-error/30251/8 "2019-10-24T05:31:27Z")

</div>

> [@natgeo-wong](#):
>
> Does anyone have any ideas as to why this could happen? Julia works just fine on my personal computer, but I’d rather not download 2-3 years worth of data on my computer and leave it overnight.

Is it the same version of Julia and the packages on both computers?

I am assuming the server has more RAM than your PC and it is not limited per user/process to a smaller number. You may want to check these things, eg on a Linux server with `ulimit` and `cgroups`, or ask the server admins.

Other than this, you could track free memory when debugging, eg

```julia
julia> round(Int, Base.Sys.free_memory() / 2^20)
8027

julia> A = ones(1000, 1000, 1000);

julia> round(Int, Base.Sys.free_memory() / 2^20)
1966

```

You should then make sure that objects which are no longer used [can be garbage collected](https://docs.julialang.org/en/v1/manual/faq/#How-do-I-delete-an-object-in-memory?-1).

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [October 24, 2019, 9:21am UTC](https://discourse.julialang.org/t/memory-error/30251/9 "2019-10-24T09:21:05Z")

</div>

As @tamas_papp says you need to look at the limits for your user session on this server.  
Could you please tell us more about the server - which OS It is running, how much RAM it has

cat /etc/_elease_  
uname -r  
free -g

Can we assume you are starting an ssh session on the server, then running ‘screen’ or something similar?

---

<div class="post-metadata">

**Author:** ![natgeo-wong](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/natgeo-wong/32/10965_2.png) [@natgeo-wong](https://discourse.julialang.org/u/natgeo-wong)\
**Post date:** [October 24, 2019, 1:06pm UTC](https://discourse.julialang.org/t/memory-error/30251/10 "2019-10-24T13:06:16Z")

</div>

Okay, after going an entire circle, it was found that the memory error is due to NetCDF’s memory leakage ahha. Thank you to everyone for your responses!