# Memory issues

**URL:** <https://discourse.julialang.org/t/memory-issues/37594>\
**Category:** Performance\
**Created:** [April 14, 2020, 10:29pm UTC](https://discourse.julialang.org/t/memory-issues/37594 "2020-04-14T22:29:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![natgeo-wong](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/natgeo-wong/32/10965_2.png) [@natgeo-wong](https://discourse.julialang.org/u/natgeo-wong)\
**Post date:** [April 14, 2020, 10:29pm UTC](https://discourse.julialang.org/t/memory-issues/37594/1 "2020-04-14T22:29:35Z")

</div>

Hi! Got a question on memory handling:

For the following script

```julia
nlon = 360; nlat = 181; nhr = 744; np = 37;

for loop = 1 : 5

    @info "$(Dates.now()) - Preallocating arrays ..."
    Ta = zeros(nlon,nlat,nhr,np); sH = zeros(nlon,nlat,nhr,np);
    za = zeros(nlon,nlat,nhr,np); Tm = zeros(nlon,nlat,nhr);

    @info "$(Dates.now()) - Extracting Surface-Level data for $(dtii) ..."
    Ts = rand(nlon,nlat,nhr); Td = rand(nlon,nlat,nhr)

    @info "$(Dates.now()) - Extracting Pressure-Level data for $(dtii) ..."
    for pii = 1 : np; pre = p[pii];
        Ta[:,:,:,pii] = rand(nlon,nlat,nhr)
        sH[:,:,:,pii] = rand(nlon,nlat,nhr)
        za[:,:,:,pii] = rand(nlon,nlat,nhr)
    end

end

```

After a loop or two, my job gets killed on a cluster because the amount of memory required is too large. However, I requested 80GB of memory, so honestly I am not sure what the issue is here. Is there something I’m missing in memory handling in `for`-loops?

For reference `360*181*37*24*37*3 = 6.5*10^9` which in Float64 is around 51 GB, so I’m not too sure what the issue is here because it definitely is enough memory for at least 1 loop (which is true, it does work fine for the first loop or two). But I’m just not sure if the memory carries over into next iterations. Why would this be the case?

Edit: Below is the output:

```julia
[nwong@holy7c26401 scripts]$ julia test2.jl 
[ Info: 2020-04-14T18:21:27.308 - Preallocating arrays ...
[ Info: 2020-04-14T18:21:40.976 - Extracting Surface-Level data for 1 ...
[ Info: 2020-04-14T18:21:41.569 - Extracting Pressure-Level data for 1 ...
[ Info: 2020-04-14T18:22:03.337 - Preallocating arrays ...
Killed

```

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [April 15, 2020, 12:19am UTC](https://discourse.julialang.org/t/memory-issues/37594/2 "2020-04-15T00:19:01Z")

</div>

Why are you zeroing out the large arrays at the top of each loop without doing anything with them? You assign to `Ta`, `sH`, and `za` in the inner loop but you don’t actually use them as far as I can see.

Other than that, what happens if you deallocate the arrays at the bottom of the loop so that the garbage collector can deal with them more quickly? (At the risk of introducing type instability, perhaps a `Ta = sH = za = Tm = []` at the bottom of the outer loop will do it.)

---

<div class="post-metadata">

**Author:** ![natgeo-wong](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/natgeo-wong/32/10965_2.png) [@natgeo-wong](https://discourse.julialang.org/u/natgeo-wong)\
**Post date:** [April 15, 2020, 12:36am UTC](https://discourse.julialang.org/t/memory-issues/37594/3 "2020-04-15T00:36:33Z")

</div>

It’s not put into the code because it’s not an important part of the code, but the results from `Ta`, `sH`, etc. are all part of a calculation that is later saved into a file with each loop, and not included here because the problem is the memory.

i have tried deallocating the arrays by doing `Ta = []` individually and doing `GC.gc()` but no luck as well.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [April 15, 2020, 6:47am UTC](https://discourse.julialang.org/t/memory-issues/37594/4 "2020-04-15T06:47:57Z")

</div>

> [@natgeo-wong](#):
>
> is around 51 GB, so I’m not too sure what the issue is here because it definitely is enough memory for at least 1 loop (which is true, it does work fine for the first loop or two).

You need a bit more, garbage collection is not immediate so new and old arrays can coexist.

Generally, I would try making operations in-place, eg

1. generate `Ta` & friends _outside_ the loop, and zero them out with `.=` _inside_,
2. use `rand!` on the whole array or a view.