Distributed.jl Processes not Shutting Down until Walltime on PBS

vijeycreative
Thu, 17 Nov 2022 07:13:34 +0000
Julia at Scale

Hi All,

This is my first time posting here, and I am posting after having read through a number of similar posts on Julia Discourse. My problem in a nutshell is that when I run a script on PBS, where the script does some parallel processing, the script doesn’t terminate when all the processes have finished executing. I have tried using interrupt() and rmprocs(), and they do terminate all the processes except process 1, which continues to exist doing nothing until PBS force quits the script after exceeding the time limit.

Here is a sample code that I’m running on PBS HPC

using Distributed
addprocs(length(Sys.cpu_info()))

@everywhere begin

DO SOMETHING PARALLEL

end

println(Result from above computation)

rmprocs(length(Sys.cpu_info()))
exit()

and here is my PBS Script

#!/bin/bash
#PBS -N simulation
#PBS -l select=1:ncpus=64
#PBS -l walltime=1:00:00
# Setup Julia
export PATH="$PATH:/home/users/scratch/julia-1.8.2/bin"
julia main.jl

What baffles me is that I can’t seem to reproduce this on my local machine or on any other servers. This only occurs when running the script on PBS. If anyone has experienced similar to this and has any suggestions and alternatives, they would be much appreciated.

8 posts - 3 participants

Read full topic