Are you perhaps using GLM? I have a discussion where I found excessive usage in julia recently (I don’t know when it started, but it wasn’t there when I originally developed the script) and it looked like GLM was using too many threads for regression. I guess it uses LinearAlgebra under the covers. I fixed it by reducing the BLAS thread count.
You can see the discussion here: Huge CPU load when using GLM
I found the solution.
I know we don’t use julia the right way. We run every script in the global environment (but we wanna change it soon)
BUT - the solution was:
I put
import Pkg
Pkg.activate()
at the very beginning of each script.
Thanks for all the help
Edit:
Not sure if the obove was the solution.
I just found out that JULIA_NUM_THREADS=4
was also needed.
I find this solution quite disturbing as it seems to imply that packages may interact in unknown ways to slow down the run.
3 Likes
Probably just different versions of packages being used.
1 Like
And would that justify a “much slower than”?
I guess those versions are slower than the other ones. Shouldn’t be too hard to verify, run the code with the same Project + Manifest on different julia versions and see if there is a perf difference.
But I’m very much interested why the change in my last edit works. I run 6 scripts in parallel and wanna give them as much CPU as possible. Why do I need to restricht it to less than the system can offer?