we are experiencing extremely high first call latency times with Julia 1.6 when running our geophysical flows multiphysics finite element PDE solver (on the order of 15 hours on a supercomputer node with 192 GBytes of mem; on a local laptop it is killed after some hours by the OS due to memory consumption). Obviously, this significantly geopardizes development.
On the other hand, it is surprising that with Julia 1.5 the first call latency of the very same code is only around 15 mins on the local laptop referred above.
For the records, the code is available in the public domain in the following git repo
And can be executed as
git clone https://github.com/gridapapps/GridapGeosciences.jl cd GridapGeosciences.jl git checkout remove_dependency_pardiso julia --project=. using Pkg; Pkg.instantiate() include("test/Williamson2ThetaMethodFullNewtonTests.jl")
We would like to understand the cause of the problem. It seems that some feature(s) in our code are challenging the Julia compiler performance, but it is really hard to say without having a deep understanding of the underlying machinery. It would be ideal to come up with a MWE, but the code complexity and the fact that it stalls with Julia 1.6 does not facilitate the task.
Is there anyone in the community experiencing similar issues? Which performance analysis tools may we use to detect the cause of the problem? We guess that this issue might be related to How to cut down compile time when inference is not the problem?, but we are not certain.
Thanks in advance for your help!