# Help reducing compilation and inference time

**URL:** <https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630>\
**Category:** Performance\
**Tags:** compilation, latency\
**Created:** [July 4, 2024, 2:32pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630 "2024-07-04T14:32:51Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 4, 2024, 2:32pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/1 "2024-07-04T14:32:51Z")

</div>

We have a simulator in our company that’s fully written in Julia. The codebase is quite large. We have been suffering from very high latency for some time now. I took the initiative of trying to solve the latency issue but after working on it for a few months now, I have hit a roadblock a need some help because the results I am seeing do not make sense to me. The TTFX for our code is 4 to 6 minutes. After that, depending on the simulation parameters, the simulation can take less than a second to run.

I will explain our code using pseudo-julia-code since this is proprietary code. Here it goes!

```julia
function main(input_param_file)
    params = read_params(input_param_file)

    result_a, result_b, result_c = run_sim(params)

    print_res_summary_a(result_a)
    print_res_summary_b(result_b)
    print_res_summary_c(result_b)
end

function run_sim(params)
    # Extract params
    param_a = get_param_a(params)
    param_b = get_param_b(params)
    param_c = get_param_c(params)

    # Generate data based on params
    data = gen_data(param_a, param_b, param_c)

    # Loop
    while some_condition
        # main processing functions
        res_a = process_a(param_a, data)
        res_b = process_b(param_b, data)
        res_c = process_c(param_c, data)

        # Write results to files
        dump_results(res_a, res_b, res_c)
    end

    return res_a, res_b, res_c
end

```

I made sure that `main`, `run_sim` and the functions it calls are type-stable. I timed the **first call** of almost all of the functions above. This is using a reference param file. Here are the results:

| function | time (seconds) |
| --- | --- |
| main | 229.8 |
| run\_sim | 8.5 |
| process\_a | 0.58 |
| process\_b | 9E-06 |
| process\_c | 5E-06 |
| dump\_results | 0.0029 |

As you can see, all of the functions have decent compilation/inference times with the exception of `main` and I don’t know why.

Precompiling is an option but it does not solve the fundamental issue we are having. Also, this will be our last resort.

I used `SnoopCompile` to get a sense of the amount and impact of runtime inferences. While there are runtime inferences (~150), as far as I can tell, they don’t seem to be affecting the latency that much.

Any help will be deeply appreciated. I will try to provide more information if needed given the proprietary nature of our code.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [July 4, 2024, 2:45pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/2 "2024-07-04T14:45:19Z")

</div>

Could you maybe provide a minimal _reproducible_ example?

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 4, 2024, 2:49pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/3 "2024-07-04T14:49:08Z")

</div>

I wish I could. If I do, I will probably get fired within the next few hours 🙂

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [July 4, 2024, 2:59pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/4 "2024-07-04T14:59:16Z")

</div>

I understand, I’m just not sure what we can do here. You’re probably already aware of relevant tools: beyond SnoopCompile.jl, Cthulhu.jl might help to diagnose faulty inference?

> I made sure that `main`, `run_sim` and the functions it calls are type-stable.

How did you ensure it? Just by checking the output inference, or with more sophisticated checks like JET.jl?

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 4, 2024, 3:04pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/5 "2024-07-04T15:04:10Z")

</div>

I used Cthulhu and SnoopCompile extensively to analyze our code. I fixed some of the inference problems but the large TTFX of main does not improve.

Can you can tell me what I should look for in the SnoopCompile results?

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [July 4, 2024, 3:05pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/6 "2024-07-04T15:05:42Z")

</div>

Have you profiled the first call to `main` with `@profview` to make sure that the time is indeed spent compiling/inferring and not, say, reading the file?

> Can you can tell me what I should look for in the SnoopCompile results?

Unfortunately I have next to zero experience with that one

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 4, 2024, 3:06pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/7 "2024-07-04T15:06:50Z")

</div>

> How did you ensure it? Just by checking the output inference, or with more sophisticated checks like [JET.jl](https://juliahub.com/ui/Packages/General/JET)?

I used Cthulhu to identify type-unstable functions and fixed them by making sure the output types are deterministic given a set of input types.

---

<div class="post-metadata">

**Author:** ![TimG](https://avatars.discourse-cdn.com/v4/letter/t/82dd89/32.png) [@TimG](https://discourse.julialang.org/u/TimG)\
**Post date:** [July 4, 2024, 3:23pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/8 "2024-07-04T15:23:48Z")

</div>

Is it possible you are seeing a manifestation of this issue?

[https://discourse.julialang.org/t/investigating-large-latency-on-a-constrained-windows-environment/114482](https://discourse.julialang.org/t/investigating-large-latency-on-a-constrained-windows-environment/114482)

You describe your problem quite differently so probably not but, just in case…

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 4, 2024, 3:28pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/9 "2024-07-04T15:28:52Z")

</div>

This looks more like a third-party package issue. We don’t use heavy third-party packages in our code

---

<div class="post-metadata">

**Author:** ![sbuercklin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sbuercklin/32/15728_2.png) [@sbuercklin](https://discourse.julialang.org/u/sbuercklin)\
**Post date:** [July 4, 2024, 3:29pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/10 "2024-07-04T15:29:40Z")

</div>

> [@osimonn](#):
>
> I timed the **first call** of almost all of the functions above.

In the same Julia instance, or restarting between each timing? If you don’t restart, then `run_sim` will get compiled by your call to `main` so it won’t have any compilation overhead.

And how long do these take on subsequent calls, so we know how much is compilation vs how much is runtime?

---

<div class="post-metadata">

**Author:** ![fatteneder](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fatteneder/32/33991_2.png) [@fatteneder](https://discourse.julialang.org/u/fatteneder)\
**Post date:** [July 4, 2024, 3:32pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/11 "2024-07-04T15:32:48Z")

</div>

Two questions:

- Is `params` a named tuple with lots of fields?
- Do you happen to use the splat operator (`...`) extensively?

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 4, 2024, 3:32pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/12 "2024-07-04T15:32:50Z")

</div>

> In the same Julia instance, or restarting between each timing?

Restarting each time.

> And how long do these take on subsequent calls, so we know how much is compilation vs how much is runtime?

Less than a minute. It is negligible compared to first call.

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 4, 2024, 3:34pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/13 "2024-07-04T15:34:41Z")

</div>

> Is `params` a named tuple with lots of fields?

`params` is a dictionary and gets mapped to structs in the `get_params_*` functions

> Do you happen to use the splat operator (`...`) extensively?

We do use the splat operator but not extensively.

---

<div class="post-metadata">

**Author:** ![sbuercklin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sbuercklin/32/15728_2.png) [@sbuercklin](https://discourse.julialang.org/u/sbuercklin)\
**Post date:** [July 4, 2024, 3:35pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/14 "2024-07-04T15:35:24Z")

</div>

What is the timing of compiling `read_params`? It looks like your `run_sim` function is fast to compile + run, so it has to be something else in `main`

---

<div class="post-metadata">

**Author:** ![fatteneder](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fatteneder/32/33991_2.png) [@fatteneder](https://discourse.julialang.org/u/fatteneder)\
**Post date:** [July 4, 2024, 3:37pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/15 "2024-07-04T15:37:06Z")

</div>

> [@osimonn](#):
>
> > Is `params` a named tuple with lots of fields?
> 
> `params` is a dictionary and gets mapped to structs in the `get_params_*` functions

Presumably these getters are not performance critical, did you try to annotate their signatures using `@nospecialize`?  
E.g.

```julia
function get_param_a(@nospecialize(params))
   ...
end

```

This would prevent excessive code generation, if that’s the bottleneck (although not sure).

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 4, 2024, 3:38pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/16 "2024-07-04T15:38:21Z")

</div>

That’s what I thought at first but I did an experiment where I removed `main` and ran `run_sim` without `main`. The latency shifted to `run_sim`, i.e. `run_sim` took roughly 229.8 seconds to start. Same as `main`

---

<div class="post-metadata">

**Author:** ![osimonn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/osimonn/32/44888_2.png) [@osimonn](https://discourse.julialang.org/u/osimonn)\
**Post date:** [July 4, 2024, 3:40pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/17 "2024-07-04T15:40:00Z")

</div>

`params` is of type `Dict{Symbol,Any}`. Will `@nospecialize` make a difference in this case?

---

<div class="post-metadata">

**Author:** ![fatteneder](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fatteneder/32/33991_2.png) [@fatteneder](https://discourse.julialang.org/u/fatteneder)\
**Post date:** [July 4, 2024, 3:41pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/18 "2024-07-04T15:41:39Z")

</div>

Hmm, nah, probably not. Sorry.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [July 4, 2024, 3:46pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/19 "2024-07-04T15:46:15Z")

</div>

Is your code in a package? They are precompiled to machine code, and that’s the only way I know how to do that with the exception of putting your code in the sysimage (it’s not to complex, has pros and cons), another option. Or compiling to an app, with PackageCompiler.jl which is implemented that way, I believe.

> [@osimonn](#):
>
> The TTFX for our code is 4 to 6 minutes.

That is quite long… I understand you don’t want to wait (but curious, is it a large fraction of the simulation time, how long in total?). You can also use **[DaemonMode.jl](https://github.com/dmolina/DaemonMode.jl)** is that applies to you, to not pay for TTFX (for scripts).

Python packages are also precompiled (there to bytecode only, and in latest beta version it has JIT). I would like that modules, not just packages in Julia were precompiled, and even just scripts, were precompiled, but that does not happen even with Python I think. I.e. you don’t have make and to register a full package to get precompilation… (or other manual steps). You CAN have a local package, i.e. in a local registry, not the public General registry, but I’ve not tried to set one up, seems also like a hoop to jump through.

---

<div class="post-metadata">

**Author:** ![sbuercklin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sbuercklin/32/15728_2.png) [@sbuercklin](https://discourse.julialang.org/u/sbuercklin)\
**Post date:** [July 4, 2024, 3:47pm UTC](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630/20 "2024-07-04T15:47:53Z")

</div>

How is this timing different from the timing you reported for `run_sim` in the OP?

This sounds like the problem is actually internal to `run_sim`. At that point, you need to time things like `process_a, process_b, process_c` from scratch similar to how you described timing `run_sim` without calling it from `main`

[Next page](https://discourse.julialang.org/t/help-reducing-compilation-and-inference-time/116630.md?page=2)