A short question. Is there some way to use pmap such that the following doesn’t throw an exception and crash? I have a distributed system, where a process might use too much memory and be killed by the os. This seems to result in a ProcessExitedException and the entire program crashing. I am, however, okay with individual batches failing and would like the program to continue.
using Distributed
addprocs(3)
@everywhere function calc(i)
val = mod(first(rand(Int, 1)), 4) + 1
sleep(val)
if myid() == 2 # Emulating Process Killed due to too much memory usage
exit(1)
end
return i + val
end
result = pmap(calc, vcat(1, workers());
distributed=true,
batch_size=1,
retry_delays=[],
retry_check=nothing,
on_error=ex->-1
)
print("Didn't crash")