I am using a PBS cluster system where I request a number of CPU cores for a job. However, my Julia program seems to use more than what I allocate, causing the job to be killed by the manager!
I have tried setting JULIA_NUM_THREADS and JULIA_CPU_THREADS each to 1 to no avail - the process continues to use more than 1 CPU core, with usage reported by top to be 300-400%.
What else can I do to limit the CPU usage/cores of a Julia process or Julia code?
I am using Julia v1.6.1, but can update if needed.
This might not be relevant, but the process I am running uses QSM.jl.
Maybe try setting the number of BLAS threads explicitly: OPENBLAS_NUM_THREADS=1 (or whatever value makes sense for your application). Also, you should be able to use htop to see which threads inside of the julia process are causing this.
I would have expected though that PBS sets affinity settings such that you can’t use CPU-cores outside of your allocation in the first place. But I guess this varies a lot between scheduling systems and clusters. Generally, I tend to only use nodes exclusively and manually tune everything to fully utilize the node to avoid any such node-sharing hassles.