# Benchmarking and Pkg.test()

**URL:** <https://discourse.julialang.org/t/benchmarking-and-pkg-test/27719>\
**Category:** Performance\
**Tags:** pkg, benchmarktools\
**Created:** [August 19, 2019, 3:49pm UTC](https://discourse.julialang.org/t/benchmarking-and-pkg-test/27719 "2019-08-19T15:49:39Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [August 19, 2019, 3:49pm UTC](https://discourse.julialang.org/t/benchmarking-and-pkg-test/27719/1 "2019-08-19T15:49:39Z")

</div>

This post is an offspring of the following discussion, where I reported inconsistencies in the benchmarked performances of `Base.sum` between machines.

> [@Julia equivalent of Python's "fsum" for floating point summation](https://discourse.julialang.org/t/julia-equivalent-of-pythons-fsum-for-floating-point-summation/17785/50):
>
> This is so great. slight_smile That’s entirely possible. Newer chips have much more capable SIMD vectorization units. The newest instruction sets are AVX2 (most new laptop chips) and AVX512 (only on beefy Xeon server chips). You can check which instruction sets your chips implement in the “flags” fields of /proc/cpuinfo (on Linux) or sysctl -a | grep machdep.cpu.features (on macOS).

To summarize the issue (which is illustrated a bit more clearly below) while benchmarking summation algorithms as compared to `Base.sum`, I collected a few results coming from various colleagues’ machines. And noticed large variations in the benchmarked performances of `Base.sum` across machines

I suggested that this might have to do with vectorization, and @mbauman provided ways to check whether the SSE/AVX/AVX2/AVX512 capabilities of the CPU explained these differences.

It turns out that instead, these variations had to do with how these benchmarks were run: from a standalone call of the julia compiler, or _via_ `Pkg.test()`

Here is a very simple example of a mostly empty package, in which the `test/runtests.jl` file has the following contents:

```julia
using BenchmarkTools
@btime sum($(rand(1_000)))

```

Then, we get: when running the test file from the command line:

```julia
> julia --project -O3 test/runtests.jl 
  84.877 ns (0 allocations: 0 bytes)

```

but when running from `Pkg.test()`:

```julia
shell> julia --project --quiet -O3
julia> using Pkg; Pkg.test()
   Testing BenchSum
 Resolving package versions...
  795.624 ns (0 allocations: 0 bytes)
   Testing BenchSum tests passed 

```

In case anyone is wondering, the situation is almost exactly the same without the `-O3` flag.

That’s nearly a 10x slow-down! Is it expected? Does anyone have an idea why this happens?

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [August 19, 2019, 3:51pm UTC](https://discourse.julialang.org/t/benchmarking-and-pkg-test/27719/2 "2019-08-19T15:51:12Z")

</div>

Tests run with `--check-bounds=yes`.

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [August 19, 2019, 4:03pm UTC](https://discourse.julialang.org/t/benchmarking-and-pkg-test/27719/3 "2019-08-19T16:03:14Z")

</div>

Thanks! That explains everything.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 19, 2019, 4:06pm UTC](https://discourse.julialang.org/t/benchmarking-and-pkg-test/27719/4 "2019-08-19T16:06:35Z")

</div>

I check generated IR in tests and realized that other flags like `--code-coverage` changes the result as well. The workaround I’ve been using is to launch a subprocess without problematic flags. Since doing this in each package is tedious, I created a helper package that does it: [Home · IRTest.jl](https://tkf.github.io/IRTest.jl/dev/) (Just FYI)

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [February 19, 2026, 11:35am UTC](https://discourse.julialang.org/t/benchmarking-and-pkg-test/27719/5 "2026-02-19T11:35:36Z")

</div>

Sorry reviving this old thread, but this is the exact same question. It is possible to disable `--check-bounds=yes` selectively for a test?

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [February 19, 2026, 2:18pm UTC](https://discourse.julialang.org/t/benchmarking-and-pkg-test/27719/6 "2026-02-19T14:18:28Z")

</div>

One easy way to do that is to just launch a new Julia session. See, e.g., Julia’s [`test/boundscheck.jl`](https://github.com/JuliaLang/julia/blob/master/test/boundscheck.jl), which runs the [`test/boundscheck_exec.jl`](https://github.com/JuliaLang/julia/blob/master/test/boundscheck_exec.jl) file with the three possible configurations.