# Help reducing compilation and inference time

**URL:** <https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630>\
**Category:** Performance\
**Tags:** compilation, latency\
**Created:** [July 4, 2024, 2:32pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630 "2024-07-04T14:32:51Z")\
**Posts on this page:** 12\
**Page:** 4

<div class="post-metadata">

**Author:** ![mrufsvold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrufsvold/32/31600_2.png) [@mrufsvold](https://discourse.julialang.org/u/mrufsvold)\
**Post date:** [July 6, 2024, 10:25am UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/61 "2024-07-06T10:25:37Z")

</div>

This is something I want to do, but haven’t had time to work on: have a nightly CI run that builds an image of the current state of the Manifest.toml and builds a sysimage. Tag the image with the hash of the manifest file. Then each CI run, checks the Manifest file against the base image tag, if it matches, it can skip installing and recompiling dependencies.

---

<div class="post-metadata">

**Author:** ![mrufsvold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrufsvold/32/31600_2.png) [@mrufsvold](https://discourse.julialang.org/u/mrufsvold)\
**Post date:** [July 6, 2024, 10:40am UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/62 "2024-07-06T10:40:16Z")

</div>

I don’t know what your technical limitations are to pre compilation. I also don’t know what you mean by having a compiled directory causing issues for other teams, so I can’t propose a good solution.

But from my read of this thread, it sounds like you’re near the upper bound of how fast Julia can compile your code base from scratch, so this becomes a DevOps problem.

The approach I described in my comment above might be applicable here. If you could build a sysimage with all your static dependencies that runs as many library methods as possible overnight, and then, have your CI/CD pipeline pull that in, you wouldn’t have to pay so much compilation time.

It makes it a bit complicated because you need to ensure that your CI can get the base image that matches the current state of your dependencies, But that’s a tractable problem.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [July 6, 2024, 9:44pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/63 "2024-07-06T21:44:27Z")

</div>

> [@osimonn](#):
>
> What are valleys in the middle?

I suspect this is measuring the backend (mostly LLVM) of the compiler, which does not show up in a lot of profiles.

> [@osimonn](#):
>
> In both case, our code (the right-hand-side cluster) takes a relatively short time to compile and execute

A slight clarification. I believe the left cluster is actually measuring the compilation of your code, whereas the right one is measuring the runtime. That’s why you see functions related to type inference and whatnot on the left. The profile is likely compiler front + middle end (left cluster) → compiler backend (middle valley) → runtime (right cluster).

It’s notable that running with `-O0` removes so much of the middle area. Have you checked if TTFX is faster? If it is, then it’s worth looking into which functions are giving LLVM a hard time. I can provide general tips such as not `@inline`ing every function and avoiding large tuples, but it would be a good idea to see what people who have experience troubleshooting this stuff have to say.

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 7, 2024, 12:58pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/65 "2024-07-07T12:58:39Z")

</div>

> [@ToucheSir](#):
>
> Have you checked if TTFX is faster?

Yes. Here are some crude results.  
Without an opt flag: 4m4.478s  
With `-O0`: 1m34.976s

> [@ToucheSir](#):
>
> it’s worth looking into which functions are giving LLVM a hard time

That’s precisely what I want to achieve but I don’t know how or where to start from.

> [@ToucheSir](#):
>
> not `@inline`ing every function

We use `@inline` in our code but not extensively. Additionally, we usually use the macro for small functions. I will remove the inline macros and see if it will make a difference but I have my doubts.

> [@ToucheSir](#):
>
> avoiding large tuples

We do have a lot of large tuples and nested tuples because we are interfacing with C code. However, I thought this issue was fixed in Julia 1.9. I posted about this issue a couple of years ago [here](https://discourse.julialang.org/t/reduce-compilation-time-for-long-and-nested-tuples/91392). Julia 1.9 did indeed reduce our compilation time significantly but the compilation time still remains high as you can tell by the aforementioned reported times at the beginning of this reply.

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 8, 2024, 2:20pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/66 "2024-07-08T14:20:14Z")

</div>

While I cannot generate flamegraphs from SnoopCompile. Here are some results that could be helpful in this investigation:

```julia
julia> tinf = @snoopi_deep main("input", "another_input")
julia> tinf
InferenceTimingNode: 169.692228/230.404140 on Core.Compiler.Timings.ROOT() with 196 direct children
julia> julia> sort(tinf.children; lt=(x, y) -> x.mi_timing.inclusive_time < y.mi_timing.inclusive_time, rev=true)[1:3]
InferenceTimingNode: 0.007097/57.119113 on main(::String, ::String) with 17 direct children
InferenceTimingNode: 0.002234/0.553512 on some_function1 with 2 direct children
InferenceTimingNode: 0.001673/0.253361 on some_function2 with 1 direct children

```

What I am gathering from this information is that `main` is the main bottleneck while other functions’ compilation times are relatively negligible.

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 8, 2024, 2:27pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/67 "2024-07-08T14:27:23Z")

</div>

Additional SnoopCompile results:

```julia
julia> tinf = @snoopi_deep main("input", "another_input")
julia> itrigs = inference_triggers(tinf)
julia> x = sort(itrigs; lt=(x, y) -> x.node.mi_timing.inclusive_time < y.node.mi_timing.inclusive_time, rev=true)
julia> x[1].node.mi_timing.inclusive_time
0.5535115639999997

```

The worst runtime inference call in terms of timing is not that bad.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [July 9, 2024, 2:19pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/68 "2024-07-09T14:19:44Z")

</div>

> [@osimonn](#):
>
> We do have a lot of large tuples and nested tuples because we are interfacing with C code. However, I thought this issue was fixed in Julia 1.9.

To my knowledge, this is still a problem. As evidenced by the regular posts of someone experiencing slowdowns with large StaticArrays.

It is suspicious that `main` only accounts for 57/169 seconds of inference time according to SnoopCompile. Perhaps worth digging into. I was hoping some folks would comment after sharing this on Slack, but no such luck. If you keep hearing nothing, you may want to try posting there yourself. There’s a dedicated `#ttfx` channel alongside the usual help ones.

---

<div class="post-metadata">

**Author:** ![JamesNZ](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jamesnz/32/35361_2.png) [@JamesNZ](https://discourse.julialang.org/u/JamesNZ)\
**Post date:** [July 9, 2024, 2:43pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/69 "2024-07-09T14:43:26Z")

</div>

For profiling compilation I think you’ll need to use Tracy with a custom build of Julia: [External Profiler Support · The Julia Language](https://docs.julialang.org/en/v1/devdocs/external_profilers/)

Though I’ve never used it myself so I can’t help much there. You’ve probably already tried this, but the only other thing I can think of would be a process of elimination, i.e. comment out parts of `main()`/`run_sim()` until you find a bottleneck. I’m guessing you’re either hitting some pathological case or death by a thousand cuts 😅

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 9, 2024, 2:58pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/70 "2024-07-09T14:58:51Z")

</div>

> [@JamesNZ](#):
>
> you’ll need to use Tracy with a custom build of Julia

I can try Tracy even though it looks like a headache to get it to work and sift through all the results 😢

> [@JamesNZ](#):
>
> You’ve probably already tried this, but the only other thing I can think of would be a process of elimination

I have tried this a couple of times and the compilation time savings were minimal but it might worth trying it out different permutations.

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 9, 2024, 3:03pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/71 "2024-07-09T15:03:13Z")

</div>

> [@ToucheSir](#):
>
> To my knowledge, this is still a problem. As evidenced by the regular posts of someone experiencing slowdowns with large StaticArrays.

I can try removing most of the code that use large tuples and see if it will make a difference. It will take some time.  
Another reason why I am doubting the large tuples theory (I could be wrong) is because, if you look at the cartoon code in my OP, almost all of the large tuples are created and processed in `process_a`, `process_b` and `process_c`. The `process_*` functions do not take a long time to compile. So how can the large tuple compilation times be shifted to the `main` function?

> [@ToucheSir](#):
>
> I was hoping some folks would comment after sharing this on Slack, but no such luck.

I appreciate you posting this issue on Slack. Thank you. I don’t have a Slack account but I can create one and post there myself.

---

<div class="post-metadata">

**Author:** ![sbuercklin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sbuercklin/32/15728_2.png) [@sbuercklin](https://discourse.julialang.org/u/sbuercklin)\
**Post date:** [July 11, 2024, 11:48am UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/72 "2024-07-11T11:48:07Z")

</div>

> [@osimonn](#):
>
> The `process_*` functions do not take a long time to compile. So how can the large tuple compilation times be shifted to the `main` function?

Just to confirm, **how** did you time the compilation time of `process_*`?

Earlier you said when you ran `run_sim` directly, the compilation slowdown “shifted” to `run_sim`. That sounds to me like `main` is not the source of the compilation slowdown, rather one of the internal functions.

Depending on the types making up the tuples, you could potentially simplify inference by wrapping them in a struct with static fieldtypes (or far fewer free type parameters) and pass that to the foreign call instead. So long as the memory layout is the same between the struct with far fewer free type parameters and your Tuple with far more type parameters, you might be able to observe a speedup and still call the function.

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 11, 2024, 1:53pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/73 "2024-07-11T13:53:27Z")

</div>

> [@sbuercklin](#):
>
> Just to confirm, **how** did you time the compilation time of `process_*`?

I wrote a macro that roughly does the following:

```julia
t1 = time()
res_a = process_a(param_a, data)
t2 = time()
push!(some_global_vec, t2 - t1)

```

Then I report the first element of `some_global_vec` as the compilation time, i.e. `some_global_vec[1]`. Technically, this time is compilation + runtime but I purposefully choose simulation parameters that do not have have high runtime. Therefore, the runtime is negligible compared to the compilation time.

> [@sbuercklin](#):
>
> Depending on the types making up the tuples, you could potentially simplify inference by wrapping them in a struct with static fieldtypes (or far fewer free type parameters) and pass that to the foreign call instead. So long as the memory layout is the same between the struct with far fewer free type parameters and your Tuple with far more type parameters, you might be able to observe a speedup and still call the function.

That is an interesting proposition. Can you provide a simple example to elaborate your point?

I can give you some examples of the structs that we are dealing with:

```julia
struct Struct2
    a::UInt8
    b::UInt8
    c::UInt8
end

struct Struct4
    a::Int16
    b::Int16
end

struct Struct3
    a::NTuple{4, Struct4}
end

struct Struct7
    a::UInt8
    b::UInt8
    c::NTuple{4, UInt16}
end

struct Struct5
    a::UInt8
    b::NTuple{16, UInt8}
end

struct Struct6
    a::UInt64
    b::NTuple{2, Cfloat}
    c::Cfloat
    d::Int16
    e::Int8
    f::Bool
end

struct Struct1
    a::UInt8
    b::UInt8
    c::UInt16
    d::UInt8
    e::UInt8
    f::UInt16
    # ...
    # Some other fields
    # ...
    cc::NTuple{8, Struct7}
    dd::Struct5
    g::Ptr{UInt8}
    h::Ptr{UInt8}
    i::Struct6
    # ...
    # Some other fields
    # ...
    aa::Struct2
    bb::Bool
    cc::NTuple{4, Struct3}
end

```

We deal with more complicated structs but this should give you a rough idea.

[Previous page](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630.md?page=3)