# Extremely high first call latency Julia 1.6 versus 1.5 with multiphysics PDE solver

**URL:** <https://discourse.julialang.org/t/extremely-high-first-call-latency-julia-1-6-versus-1-5-with-multiphysics-pde-solver/71692>\
**Category:** Performance\
**Tags:** question\
**Created:** [November 17, 2021, 11:33pm UTC](https://discourse.julialang.org/t/extremely-high-first-call-latency-julia-1-6-versus-1-5-with-multiphysics-pde-solver/71692 "2021-11-17T23:33:43Z")\
**Posts on this page:** 8\
**Page:** 2

<div class="post-metadata">

**Author:** ![amartinhuertas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/amartinhuertas/32/30886_2.png) [@amartinhuertas](https://discourse.julialang.org/u/amartinhuertas)\
**Post date:** [November 26, 2021, 7:29am UTC](https://discourse.julialang.org/t/extremely-high-first-call-latency-julia-1-6-versus-1-5-with-multiphysics-pde-solver/71692/21 "2021-11-26T07:29:40Z")

</div>

> Would it be possible to simplify the code (i.e. parameters like grid size, number of equations etc.) in Julia 1.5 to run in say seconds instead of minutes? I’d expect a MWE on Julia 1.6 to run in minutes instead of hours then, which might simplify things for analysis.

There is this post. [How to cut down compile time when inference is not the problem?](https://discourse.julialang.org/t/how-to-cut-down-compile-time-when-inference-is-not-the-problem/59077) However, I do not actually know if there is actually relation among the causes underlying the observations in this thread and the one at that thread.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [November 26, 2021, 7:40am UTC](https://discourse.julialang.org/t/extremely-high-first-call-latency-julia-1-6-versus-1-5-with-multiphysics-pde-solver/71692/22 "2021-11-26T07:40:24Z")

</div>

> [@amartinhuertas](#):
>
> GridapGeosciences.jl highly relies on Gridap.jl. Thus, I wouldnt be surprised if the bulk of the compilation times are concentrated in Gridap.jl’s code triggered from GridapGeosciences.jl.

Then they might want to modify all `@noinline` methods in `SparseMatrixAssemblers.jl` adding a `@nospecialize` like so:

```julia
@noinline function _numeric_loop_vector!(vec,caches,cell_vals,cell_rows)
  @nospecialize
  add_cache, vals_cache, rows_cache = caches
  @assert length(cell_vals) == length(cell_rows)
  add! = AddEntriesMap(+)
  for cell in 1:length(cell_rows)
    rows = getindex!(rows_cache,cell_rows,cell)
    vals = getindex!(vals_cache,cell_vals,cell)
    evaluate!(add_cache,add!,vec,vals,rows)
  end
end

```

This seems to make things workable again for me on Julia 1.7.0-rc3. But be aware of this open [issue](https://github.com/JuliaLang/julia/issues/40312).

---

<div class="post-metadata">

**Author:** ![fverdugo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fverdugo/32/9446_2.png) [@fverdugo](https://discourse.julialang.org/u/fverdugo)\
**Post date:** [November 26, 2021, 7:59am UTC](https://discourse.julialang.org/t/extremely-high-first-call-latency-julia-1-6-versus-1-5-with-multiphysics-pde-solver/71692/23 "2021-11-26T07:59:43Z")

</div>

> [@goerch](#):
>
> Then they might want to modify all `@noinline` methods in `SparseMatrixAssemblers.jl` adding a `@nospecialize` like so:

Hi @goerch! I have implemented function `_numeric_loop_vector!` in Gridap.jl

To give more background, in this function call, `caches` and `cell_vals` are potentially VERY complex nested inmutable objects (so no surprise that this is a hot spot). I am curious how `@nospecialize` would affect the run time performance taking into account that `length(cell_rows)` can be of the order of 10^7 for a large finite element mesh.

> This seems to make things workable again for me on Julia 1.7.0-rc3. But be aware of this open [issue](https://github.com/JuliaLang/julia/issues/40312).

How about calling `Base.inferencebarrier` on `caches` and `cell_vals`, which are the bad guys? Do you think this can also improve things?

I am aware that we have complex types in Gridap.jl, but it was not a fatal problem until Julia 1.6 for sophisticated equations.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [November 26, 2021, 8:06am UTC](https://discourse.julialang.org/t/extremely-high-first-call-latency-julia-1-6-versus-1-5-with-multiphysics-pde-solver/71692/24 "2021-11-26T08:06:07Z")

</div>

Hi @fverdugo,

my first suspicion would be that the increased compile time is due to _new_ optimizations. So I’d expect a discussion about missed optimizations next, of course;)

Will check the example from [here](https://discourse.julialang.org/t/how-to-cut-down-compile-time-when-inference-is-not-the-problem/59077) with `@btime` next.

Edit: done.

---

<div class="post-metadata">

**Author:** ![Alexander-Barth](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alexander-barth/32/3692_2.png) [@Alexander-Barth](https://discourse.julialang.org/u/Alexander-Barth)\
**Post date:** [November 26, 2021, 12:40pm UTC](https://discourse.julialang.org/t/extremely-high-first-call-latency-julia-1-6-versus-1-5-with-multiphysics-pde-solver/71692/25 "2021-11-26T12:40:36Z")

</div>

I have an issue with compile time latency with a neural work in Julia 1.6 and 1.7-rc2 (compared to Julia 1.5) which might be related:

> [@Significant compile time latency in Flux with a GAN](https://discourse.julialang.org/t/significant-compile-time-latency-in-flux-with-a-gan/68518):
>
> I am trying to implement a [SRGAN](https://arxiv.org/abs/1609.04802) (super-resolution generative adversarial networks) in Flux (I am happy to share once it works… :-)) The generator network is composed of 5 residual blocks (function resblock below). Unfortunately the time of the first call of Zygote.pullback increases very quickly with the number of residual blocks, so that I am not able to use the 5 residual blocks as intended. I already commented out any elements-wise broadcasting (activation functions), but the issue is s…

If one uses the option -O1 the compile times are vastly reduced. When inspecting the run with callgrind to CPUs time is mostly spend in LLVM _even on the second call_ of the same function with the same arguments (using the default optimization level).

I tried to compile Julia from source changing the parameters `max_methods`, `tupletype_depth`, `tuple_splat`, `inline_tupleret_bonus` but without luck.

> <https://github.com/JuliaLang/julia/blob/master/base/compiler/types.jl#L69>

When I interrupt Julia during the excessive compilation, it seemed to be busy in the `DAGCombiner` phase of LLVM.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [November 26, 2021, 7:05pm UTC](https://discourse.julialang.org/t/extremely-high-first-call-latency-julia-1-6-versus-1-5-with-multiphysics-pde-solver/71692/26 "2021-11-26T19:05:25Z")

</div>

Interesting, I didn’t even get the idea: is this simply an(occasionally very) expensive optimization disabled in -O1 and enabled in -O2?

Edit: OK i checked this hypothesis for @amartinhuertas problem, and indeed: the original test case works in reasonable time on Julia 1.6.4 without adding `@nospecialize` if one simply reduces the optimization level to -O1!

Edit: updated [https://github.com/JuliaLang/julia/issues/43206#issuecomment-980599991](https://github.com/JuliaLang/julia/issues/43206#issuecomment-980599991)

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [November 30, 2021, 1:20pm UTC](https://discourse.julialang.org/t/extremely-high-first-call-latency-julia-1-6-versus-1-5-with-multiphysics-pde-solver/71692/27 "2021-11-30T13:20:36Z")

</div>

Trying to reduce the original MWE I stumbled about quite some type instabilities and have the distinct impression of a relation between type instabilities and longer compile times on -O2 at least under newer versions of Julia (I worked with 1.8.0 to make the best use of JET). Therefore I filed [gridapapps/GridapGeosciences.jl#28](https://github.com/gridapapps/GridapGeosciences.jl/issues/28) and [gridapapps/GridapGeosciences.jl#29](https://github.com/gridapapps/GridapGeosciences.jl/issues/29).

Can someone confirm the impression?

Edit: not a native speaker…

---

<div class="post-metadata">

**Author:** ![amartinhuertas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/amartinhuertas/32/30886_2.png) [@amartinhuertas](https://discourse.julialang.org/u/amartinhuertas)\
**Post date:** [December 2, 2021, 8:03am UTC](https://discourse.julialang.org/t/extremely-high-first-call-latency-julia-1-6-versus-1-5-with-multiphysics-pde-solver/71692/28 "2021-12-02T08:03:02Z")

</div>

For the records, we could already “solve” (bypass) the issue. See [https://github.com/JuliaLang/julia/issues/43206#issuecomment-983474073](https://github.com/JuliaLang/julia/issues/43206#issuecomment-983474073) for more details.

[Previous page](https://discourse.julialang.org/t/extremely-high-first-call-latency-julia-1-6-versus-1-5-with-multiphysics-pde-solver/71692.md?page=1)