# Unexpected OutOfMemory error on HPC

**URL:** <https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190>\
**Category:** General Usage\
**Tags:** hpc, memory, clustering\
**Created:** [April 7, 2020, 7:54pm UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190 "2020-04-07T19:54:42Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![AhmedAlreweny](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@AhmedAlreweny](https://discourse.julialang.org/u/AhmedAlreweny)\
**Post date:** [April 7, 2020, 7:54pm UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/1 "2020-04-07T19:54:42Z")

</div>

Hello,  
I am using an HPC cluster to run a code that involves solving 2 huge sparse linear systems. I was able to test the code on my windows machine with 16GB memory. I used the macro `@time` to monitor the memory allocations at the critical places (Matrix filling up, Matrix multiplication, system solving … etc).

> 0.573989 seconds (143.03 k allocations: 710.180 MiB, 22.39% gc time)  
> 0.568460 seconds (71.21 k allocations: 1.865 GiB, 0.75% gc time)  
> 9.861626 seconds (4.70 M allocations: 1.018 GiB)  
> 0.177318 seconds (275.76 k allocations: 120.454 MiB, 19.67% gc time)  
> 0.220135 seconds (82.13 k allocations: 39.772 MiB)  
> 0.005179 seconds (123 allocations: 8.219 KiB)

I was also able to optimize the number of allocations by using `dropzeros!(..)` and other features of sparse matrices. In general, the code was consuming up to 7 GB of my meomry on windows. However, when I moved to the cluster, I allocated 10 GB for the same code and I got `OutOfMemory` error. I increased it up to 20 GB and it also crashed! It is only working when I use 25 GB. I made a plot for the mem consumption in kb measured by a tool in our cluster and I got this graph.

 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/d/fd711769f8cb19f55f87d8020e8da6fee40ec527.png)

Any suggestion?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [April 7, 2020, 8:00pm UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/2 "2020-04-07T20:00:12Z")

</div>

Can you post the code that’s doing the bulk of the work?

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [April 7, 2020, 8:22pm UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/3 "2020-04-07T20:22:46Z")

</div>

This may not be your issue, but note that if you’re running with `addprocs()` on the same node, you will be copying all data to each process. This means that, if you have an 8-core node, and you run `addprocs(7)` on it, you will have 8 separate copies of your data as each process will get its own.

Without knowing more about how you set up your cluster jobs and your multiprocessing environment, it’s hard to say what the problem is.

---

<div class="post-metadata">

**Author:** ![AhmedAlreweny](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@AhmedAlreweny](https://discourse.julialang.org/u/AhmedAlreweny)\
**Post date:** [April 7, 2020, 8:38pm UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/4 "2020-04-07T20:38:08Z")

</div>

Thanks for your prompt answer. I even tried using 1 core only on a single node, and allocated 20 GB to this core only … didn’t work!

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [April 7, 2020, 8:39pm UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/5 "2020-04-07T20:39:10Z")

</div>

If you have code and more details of your setup, that would help.

---

<div class="post-metadata">

**Author:** ![AhmedAlreweny](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@AhmedAlreweny](https://discourse.julialang.org/u/AhmedAlreweny)\
**Post date:** [April 7, 2020, 9:56pm UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/6 "2020-04-07T21:56:50Z")

</div>

```julia
@time(NNN =α^(-2) * C * C_t)
@time(A__ = B * B_t + NNN)
@time(W = A__ \ b_)
@time(V = -B_t * W)
@time(η = C \ -(b_ + B * V))
@time(y = reshape(V, N, M))

```

where:  
C : Sparse matrix of dims (659934,9999) , C\_t : its transpose  
B : Sparse matrix of dims (659934,660000) , B\_t : its transpose  
α : scalar

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [April 7, 2020, 10:21pm UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/7 "2020-04-07T22:21:13Z")

</div>

… and how have you set up your distributed environment?

---

<div class="post-metadata">

**Author:** ![traktofon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/traktofon/32/591_2.png) [@traktofon](https://discourse.julialang.org/u/traktofon)\
**Post date:** [April 8, 2020, 8:05am UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/8 "2020-04-08T08:05:45Z")

</div>

Do you happen to know which metric of “mem consumption” the cluster is reporting? Is it resident set size, virtual memory reservation, or …?

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [April 8, 2020, 8:54am UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/9 "2020-04-08T08:54:29Z")

</div>

Good questions from @traktofon  
Also ask your systems managers if the job is being run within a cgroup or container.  
It looks to be, since you are allocating a certain amount of memory.

---

<div class="post-metadata">

**Author:** ![AhmedAlreweny](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@AhmedAlreweny](https://discourse.julialang.org/u/AhmedAlreweny)\
**Post date:** [April 8, 2020, 9:04am UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/10 "2020-04-08T09:04:10Z")

</div>

Yes, that’s indeed right. The way I allocate my resources is as follows:

```julia
#PBS -l walltime .... 
#PBS -l pmem= .... //memory per core, it is also possible to use mem=... which allocates mem per node

```

---

<div class="post-metadata">

**Author:** ![AhmedAlreweny](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@AhmedAlreweny](https://discourse.julialang.org/u/AhmedAlreweny)\
**Post date:** [April 8, 2020, 9:09am UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/11 "2020-04-08T09:09:20Z")

</div>

I don’t really know exactly as I don’t have access to this data in real-time. All I have is the total vmem and mem usage reported in the stdout file. One example from a job that cached is the following:

```julia
Resource List: nodes=1:ppn=1,pmem=15gb,walltime=00:05:00,neednodes=1:ppn=1
Resources Used: cput=00:02:28,vmem=10906160kb,walltime=00:02:39,mem=7072720kb,energy_used=0

```

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [April 8, 2020, 9:41am UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/12 "2020-04-08T09:41:51Z")

</div>

Really, really sorry to ask this.  
Can you cut and paste the Out of Memory error? I think this is the PBS mechanism which is terminating the job - the job here is not being run within a cgroup and it is the PBS daemon which is looking at memory use and terminating the job.

You can get a detailed log output from the PBS job log on the first compute node, but you probably have to be a root user to do this.

But I must say this does not really help us with your Julia issue of more memory use on a Linux cluster as opposed to Windows.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [April 8, 2020, 9:44am UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/13 "2020-04-08T09:44:27Z")

</div>

Can you share your PBS job submission script?  
Also on the compute nodes how many cores are there and how much memory?  
Do you know if hyperthreading is enabled - as far as I know PBS is not aware of hyperthreading.  
I could be wrong!

---

<div class="post-metadata">

**Author:** ![AhmedAlreweny](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@AhmedAlreweny](https://discourse.julialang.org/u/AhmedAlreweny)\
**Post date:** [April 8, 2020, 9:55am UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/14 "2020-04-08T09:55:10Z")

</div>

I forgot to clarify some distinction between two scenarios, the first one, when I (think) that I allocated sufficient memory based on what I tested on Windows → this shows the OutOfMemory error as:

```julia
[ Info: started timer at: 2020-04-01T15:14:48.981
ERROR: LoadError: OutOfMemoryError()
Stacktrace:
 [1] Array at ./boot.jl:404 [inlined]
 [2] spmatmul(::SparseMatrixCSC{Float64,Int64}, ::SparseMatrixCSC{Float64,Int64}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/SparseArrays/src/linalg.jl:212
 [3] *(::SparseMatrixCSC{Float64,Int64}, ::SparseMatrixCSC{Float64,Int64}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/SparseArrays/src/linalg.jl:187
 [4] LSS() at /ddn1/vol1/site_scratch/../../../..//1500/LSS.jl:130
 [5] top-level scope at /ddn1/vol1/site_scratch//../../../../1500/Main.jl:80
 [6] include at ./boot.jl:328 [inlined]
 [7] include_relative(::Module, ::String) at ./loading.jl:1105
 [8] include(::Module, ::String) at ./Base.jl:31
 [9] exec_options(::Base.JLOptions) at ./client.jl:287
 [10] _start() at ./client.jl:460
in expression starting at /ddn1/vol1/site_scratch/../../../../1500/Main.jl:80

```

I scaled sown my problem (smaller size of the linear systems), ans I was able to run this version, but with mem allocated much much higher that expected (i.e. 25GB). When I try to make the pmem smaller (i.e pmem=10gb), the scheduler automatically kills my job without showing any error massage from julia itself (2nd scenario).

The common problem is that Julia is overusing the memory in a bizarre way that leads Julia to kill itself in the first case, and leads the scheduler to kill the job when memory over-usage is detected in the 2nd case.

---

<div class="post-metadata">

**Author:** ![AhmedAlreweny](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@AhmedAlreweny](https://discourse.julialang.org/u/AhmedAlreweny)\
**Post date:** [April 8, 2020, 9:56am UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/15 "2020-04-08T09:56:53Z")

</div>

Regarding the job script, here is an example:

```julia
#!/bin/bash -l

#PBS -l nodes=1:ppn=1 
#PBS -l pmem=15gb
#PBS -l walltime=00:05:00 

module purge
module load Julia/1.3.1 
module load monitor

cd $PBS_O_WORKDIR

monitor -d 1 julia Main.jl 2 200

```

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [April 8, 2020, 11:34am UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/16 "2020-04-08T11:34:13Z")

</div>

> [@AhmedAlreweny](#):
>
> ERROR: LoadError: OutOfMemoryError()

I do not think that user limits are the problem here. You could put this in the start of the job script to check:  
ulimit -a

Also put this in your job scrit at the start:  
free  
sysctl -a | grep mem

I am definitely leading everyone up a wrong path here. However the memory overcommit may be DISABLED on an HPC cluster - for good reasons.

---

<div class="post-metadata">

**Author:** ![AhmedAlreweny](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@AhmedAlreweny](https://discourse.julialang.org/u/AhmedAlreweny)\
**Post date:** [April 8, 2020, 12:30pm UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/17 "2020-04-08T12:30:58Z")

</div>

Alright! This is what I got (alongside with some permission denied error massages in the stderr, I don’t think I have the right to use sysctrl).

```julia
core file size (blocks, -c) 62500
data seg size (kbytes, -d) 26214400
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 770478
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) 26214400
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 770478
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
              total used free shared buff/cache available
Mem: 197734444 7752876 169395028 308 20586540 189197472
Swap: 2097148 247784 1849364
net.core.optmem_max = 20480
net.core.rmem_default = 212992
net.core.rmem_max = 67108864
net.core.wmem_default = 212992
net.core.wmem_max = 67108864
net.ipv4.igmp_max_memberships = 20
net.ipv4.tcp_mem = 4631163	6174885	9262326
net.ipv4.tcp_rmem = 4096	87380	33554432
net.ipv4.tcp_wmem = 4096	65536	33554432
net.ipv4.udp_mem = 4633461	6177950	9266922
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
vm.lowmem_reserve_ratio = 256	256	32
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.nr_hugepages_mempolicy = 0
vm.overcommit_memory = 0

```

---

<div class="post-metadata">

**Author:** ![AhmedAlreweny](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@AhmedAlreweny](https://discourse.julialang.org/u/AhmedAlreweny)\
**Post date:** [April 8, 2020, 2:43pm UTC](https://discourse.julialang.org/t/unexpected-outofmemory-error-on-hpc/37190/18 "2020-04-08T14:43:55Z")

</div>

I found this topic that might be related …

[https://github.com/JuliaLang/julia/issues/30888](https://github.com/JuliaLang/julia/issues/30888)