OOM Killer on Linux

#1

I’m curious if anyone else has run into this.

I’m on Ubuntu and trying to run an analysis on some really large datasets (40gigs binary, 1.5billion rows, 1600 chunks) using JuliaDB. I’ve frequently run into the issue where I can’t utilize all 8 threads on my CPU because I run out of memory before then - Linux kills off the processes as that happens, and then the whole execution fails. The only solution I have is to limit myself to only 2 or 3 threads, maybe 4 if I feel lucky, or to chunk out even smaller.

Is there a solution to throttling the processes as I run out of memory rather than killing off a process 70% of the way through a 30-minute-long script, so that it finishes?

0 Likes

#2

I think not. You may try increasing the amount if swap available.

0 Likes

#3

pmap is fault tolerant, at least when working with addproc processes (not sure about when using threads). Fault tolerant `pmap` when worker goes down

0 Likes