I’m curious if anyone else has run into this.
I’m on Ubuntu and trying to run an analysis on some really large datasets (40gigs binary, 1.5billion rows, 1600 chunks) using JuliaDB. I’ve frequently run into the issue where I can’t utilize all 8 threads on my CPU because I run out of memory before then - Linux kills off the processes as that happens, and then the whole execution fails. The only solution I have is to limit myself to only 2 or 3 threads, maybe 4 if I feel lucky, or to chunk out even smaller.
Is there a solution to throttling the processes as I run out of memory rather than killing off a process 70% of the way through a 30-minute-long script, so that it finishes?