Thank you so much for the insight. It seems you are right about the 300 being a user limit. I admit, I don’t understand our setup very thoroughly as this is the first time it has become necessary for me to dig into it. Here are my ulimit -a results for reference:
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1030093
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 300
virtual memory (kbytes, -v) 31309824
I think we each have our own Linux user. When I run ps -A -o uid | sort | uniq I can see group members have their own username/uid output. When we first started encountering the thread limit issue (it was showing up for R users as well) we upped the max user processes from from 200 to 300. It didn’t seem to me that I would hit a limit of 300 as a single user precompiling in Julia, so I’m still confused about getting these OpenBLAS resource errors, especially when I have set LinearAlgebra.BLAS.set_num_threads(). Is there anything else I can pursue from the Julia side of things to resolve/avoid this issue?
I have very limited understanding of the rationale behind our shared resource settings, but I will mention your suggestion of limiting cpu time rather than user processes as a potential solution from the server side.