1.8 much slower than 1.6

Are you perhaps using GLM? I have a discussion where I found excessive usage in julia recently (I don’t know when it started, but it wasn’t there when I originally developed the script) and it looked like GLM was using too many threads for regression. I guess it uses LinearAlgebra under the covers. I fixed it by reducing the BLAS thread count.

You can see the discussion here: Huge CPU load when using GLM

I found the solution.
I know we don’t use julia the right way. We run every script in the global environment :confused: (but we wanna change it soon)

BUT - the solution was:
I put

import Pkg

Pkg.activate()

at the very beginning of each script.

Thanks for all the help

Edit:
Not sure if the obove was the solution.
I just found out that JULIA_NUM_THREADS=4 was also needed.

I find this solution quite disturbing as it seems to imply that packages may interact in unknown ways to slow down the run.

3 Likes

Probably just different versions of packages being used.

1 Like

And would that justify a “much slower than”?

I guess those versions are slower than the other ones. Shouldn’t be too hard to verify, run the code with the same Project + Manifest on different julia versions and see if there is a perf difference.

But I’m very much interested why the change in my last edit works. I run 6 scripts in parallel and wanna give them as much CPU as possible. Why do I need to restricht it to less than the system can offer?