Overuse of memory leads to computer freeze

General Usage Performance
1

I have a memory issue. On my small laptop (8Gb RAM) when I build overly big matrices I get a nice OutOfMemory error, the function is stopped and all is well.
However on my desktop (~48Gb RAM) overly big matrices will take all the memory and then just freeze my computer. My only solution is then to force-reboot which is a painful process. Anyone has an idea what’s going on and how can I solve it (of course I try to solve the matrix problem on the side but the debugging is quite complicated in this context)

2

I asked a similar question a while ago and there’s a small check you could put into your code, for debugging purposes: OutOfMemoryError() instead of allocating too much resources in the job scheduler

1 Like
3

Nice! My only problem is I think one matrix is enough to freeze my computer :laughing:.
I would actually be happy with OutOfMemoryError instead of my computer freezing :sweat_smile:

1 Like
4

If you are on Windows here you can use the Task Manager to kill the stalled process.

5

I am on Linux and everything is so slow that I cannot even access the virtual console (Ctrl+Alt+F1)

6

you could try running julia with a memory limit below the physical memory limit maybe? (i didn’t try that yet): Limit memory usage for a single Linux process - Unix & Linux Stack Exchange

1 Like
7

What I do for my scientific experiments (in Linux) is:

sudo sh -c "echo 2 > /proc/sys/vm/overcommit_memory"
sudo sh -c "echo 95 > /proc/sys/vm/overcommit_ratio"

So the SO reserves 95% of the memory for userspace processes, and if a process tries to get these last 5% the allocation fails (instead of the SO letting it happen and, when the memory pages are really requested, calling the OOM reaper to kill the process with more memory allocated). For me, this is what enabled me to actually capture OutOfMemory exceptions in Julia and then skip the runs that are unfesible.

2 Likes
8

How much swap do you have (and how much is used)? This sounds like you’re swapping your PC to death.

2 Likes
9

I have 320GB on a non-SSD hard-disk, so that sounds like the issue :laughing:
I did not imagine that using too much SWAP space would slow down my computer like that! But now that you mention it that makes sense

Thanks a lot!

10

Using too much swap can (if the wrong stuff gets swapped) slow down everything, since the CPU has to constantly fetch things from disk, stalling processes. 320GB swap is most certainly waaay too much, don’t do that.

1 Like
11

yes that was definitely the problem, I reduced the swap size to 8GB and now I get a beautiful OutOfMemory error :smiley:
You saved me hours of pain, because restarting a Julia session is slow but restarting the whole computer is even worse :laughing:
Thank you so much!

1 Like