# Unstable execution time with high standard error in Julia

**URL:** <https://discourse.julialang.org/t/unstable-execution-time-with-high-standard-error-in-julia/129295>\
**Category:** Performance\
**Tags:** question\
**Created:** [May 24, 2025, 3:16pm UTC](https://discourse.julialang.org/t/unstable-execution-time-with-high-standard-error-in-julia/129295 "2025-05-24T15:16:28Z")\
**Posts on this page:** 8\
**Page:** 2

<div class="post-metadata">

**Author:** ![PanT12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pant12/32/214053_2.png) [@PanT12](https://discourse.julialang.org/u/PanT12)\
**Post date:** [May 26, 2025, 1:31pm UTC](https://discourse.julialang.org/t/unstable-execution-time-with-high-standard-error-in-julia/129295/21 "2025-05-26T13:31:35Z")

</div>

Thanks for your reply! I pay attention to the standard deviation because it is one of the criteria I use to measure the stability of an algorithm. In fact, I also record the mean, median, and standard deviation. However, when I see such a large standard deviation, I feel very puzzled about why some cases take so much time. I need to figure out whether it’s an issue with the algorithm or the code.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [May 26, 2025, 3:22pm UTC](https://discourse.julialang.org/t/unstable-execution-time-with-high-standard-error-in-julia/129295/22 "2025-05-26T15:22:07Z")

</div>

Standard deviation is probably not a good measure for distributions that are extremely far from gaussian, especially multi-modal like here, though they can give a qualitative impression of “large variability”.

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [May 26, 2025, 3:48pm UTC](https://discourse.julialang.org/t/unstable-execution-time-with-high-standard-error-in-julia/129295/23 "2025-05-26T15:48:09Z")

</div>

> [@PanT12](#):
>
> because it is one of the criteria I use to measure the stability of an algorithm.

So you’re using wall-clock runtime as a proxy for something like “how many iteration steps did my adaptive algorithm need to go below the error threshold”?

That’s not bad for quick-and-dirty impressions and for end-to-end plausibility checks, but I would recommend logging that number directly, for many reasons (e.g. it also allows to separate improvements of the algorithm and improvements of the implementation).

---

<div class="post-metadata">

**Author:** ![PanT12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pant12/32/214053_2.png) [@PanT12](https://discourse.julialang.org/u/PanT12)\
**Post date:** [May 27, 2025, 1:52am UTC](https://discourse.julialang.org/t/unstable-execution-time-with-high-standard-error-in-julia/129295/24 "2025-05-27T01:52:14Z")

</div>

In fact, the variance of the number of the iteration is small, while the running time of the whole algorithm is relatively large. This makes me confused, so I want to solve this issue.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [May 27, 2025, 3:43am UTC](https://discourse.julialang.org/t/unstable-execution-time-with-high-standard-error-in-julia/129295/25 "2025-05-27T03:43:15Z")

</div>

Depending on the RAM you have, you can also just turn off the garbage collector during the benchmark to see if that solves the issue:

```julia
using BenchmarkTools
GC.enable(false)
@benchmark your_function()
 
GC.enable(true)

```

---

<div class="post-metadata">

**Author:** ![PanT12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pant12/32/214053_2.png) [@PanT12](https://discourse.julialang.org/u/PanT12)\
**Post date:** [May 27, 2025, 7:27am UTC](https://discourse.julialang.org/t/unstable-execution-time-with-high-standard-error-in-julia/129295/26 "2025-05-27T07:27:01Z")

</div>

Thanks for your advice! The variance becomes smaller.  
Before `GC.enable(false)`, the result is

```julia
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max): 147.334 μs … 3.006 ms ┊ GC (min … max): 0.00% … 93.81%
 Time (median): 164.062 μs ┊ GC (median): 0.00%
 Time (mean ± σ): 174.205 μs ± 131.898 μs ┊ GC (mean ± σ): 5.59% ± 6.78%

                ▁▂▃▃▁▁▁▂▂▆▅▇██▇▆▅▃▃▁                             
  ▁▁▁▁▁▁▂▂▂▂▃▄▅██████████████████████▇▇▆▆▅▅▄▃▄▃▃▃▂▂▂▂▂▂▁▂▂▁▁▁▁▁ ▄
  147 μs Histogram: frequency by time 186 μs <

 Memory estimate: 385.34 KiB, allocs estimate: 268.

```

After `GC.enable(false)`, the result is

```julia
BenchmarkTools.Trial: 10000 samples with 1 evaluation.
 Range (min … max): 153.250 μs … 1.817 ms ┊ GC (min … max): 0.00% … 0.00%
 Time (median): 182.520 μs ┊ GC (median): 0.00%
 Time (mean ± σ): 187.567 μs ± 56.816 μs ┊ GC (mean ± σ): 0.00% ± 0.00%

         ▁▃▅▇▇█▆▆▅▄▅▄▆▅▄▂▃▂                                     
  ▂▂▃▅▆▇█████████████████████▇▅▅▅▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▂▂▂▂ ▄
  153 μs Histogram: frequency by time 261 μs <

 Memory estimate: 385.34 KiB, allocs estimate: 268.

```

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [May 27, 2025, 7:42am UTC](https://discourse.julialang.org/t/unstable-execution-time-with-high-standard-error-in-julia/129295/27 "2025-05-27T07:42:12Z")

</div>

Other factors that can contribute to the variance:

- other processes that are running; close all browser windows and VSCode when benchmarking
- make sure you are not using a laptop that is running on battery, otherwise the powermangement causes variations of the performance
- if you have a CPU with slow cores and performance cores, try to make sure that only the performance cores are used. This works only on Linux: [GitHub - carstenbauer/ThreadPinning.jl: Readily pin Julia threads to CPU-threads](https://github.com/carstenbauer/ThreadPinning.jl)

---

<div class="post-metadata">

**Author:** ![PanT12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pant12/32/214053_2.png) [@PanT12](https://discourse.julialang.org/u/PanT12)\
**Post date:** [May 28, 2025, 8:25am UTC](https://discourse.julialang.org/t/unstable-execution-time-with-high-standard-error-in-julia/129295/28 "2025-05-28T08:25:31Z")

</div>

I see. Thanks for your suggestion!

[Previous page](https://discourse.julialang.org/t/unstable-execution-time-with-high-standard-error-in-julia/129295.md?page=1)
