Best practice for dealing with "Killed" of Julia exits

Sometimes when I run a Julia script in the terminal, e.g. julia my_script.jl the program will exit with a simple Killed message, e.g:

image

Presumably, this is because I ran something very heavy and this just crashed stuff. Is there some recommendation/best practice for figuring out what caused this? Are there several possible causes for the Killed interruption, or am I probably looking at the process having run out of memory?

This almost always is an out of memory error.

4 Likes

So basically, I either run it on a machine with more memory, or figure out a way fo it to use less memory?

Probably option 2. Unless you were running it on a machine with only like 8GB of memory, there is only a small chance “more memory” is going to solve the problem.

1 Like

Thanks a lot, this is very helpful :slight_smile:

OK, but why not provide a bit more information in the error message?
I guess the Julia process is sent a signal to “die”. Could this signal be reported?

1 Like

Yeah, this was my problem. Usually, Julia provides errors, but when this happens it just says “Killed” with no further information, giving me no good idea of how to start debugging.

Julia isn’t giving you the error message. Your OS is. I don’t believe an OOM can be handled by a program.

1 Like

At least on Linux, I had a program that was written to catch ctrl-c and respond with a menu.
Not sure if the out-of-memory signal is “catchable”, at the moment.

1 Like

Actually, Julia itself is quite capable of detecting that condition: LoadError: OutOfMemoryError() .

1 Like

Got it, but if Julia can display a LoadError: OutOfMemoryError(), is there a reason it doesn’t use it?

OutOfMemoryError can only be thrown when Julia tries to allocate and the allocation fails. You get OOMed when you write to a page that previously had no data given to it so the OS didn’t actually have any pages free even though it promised you it would.

2 Likes

Ok, got it.

I think OOM killer sends a SIGTERM signal.
Julia should be able to catch that

1 Like

When I get these, I usually throw some println statements into my program so I know roughly whereabouts it is up to at any given time, and then I run the routine in a terminal, and simultaneously keep a system monitor window (on Linux - but I’m sure there is a Windows equivalent) open next to my running program, so I can get a feel for what is causing the memory problems.

If you need to go deeper than that, then ProfileView.jl is a good place to start.

1 Like

I don’t think so, as far as I’m aware it sends a SIGKILL (which you can’t recover from). Giving control back to the program when the OS has already decided to (forcefully) stop your code doesn’t really make sense. If it were different, we’d call it the OOM hinter, not the OOM killer.

1 Like

13.4 here: Out Of Memory Management

2 Likes