# \[ANN\] AirspeedVelocity.jl - easily benchmark Julia packages over their lifetime

**URL:** <https://discourse.julialang.org/t/ann-airspeedvelocity-jl-easily-benchmark-julia-packages-over-their-lifetime/97221>\
**Category:** Package Announcements\
**Tags:** package, announcement, benchmark\
**Created:** [April 7, 2023, 1:16pm UTC](https://discourse.julialang.org/t/ann-airspeedvelocity-jl-easily-benchmark-julia-packages-over-their-lifetime/97221 "2023-04-07T13:16:36Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![MilesCranmer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/milescranmer/32/21070_2.png) [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)\
**Post date:** [April 7, 2023, 1:16pm UTC](https://discourse.julialang.org/t/ann-airspeedvelocity-jl-easily-benchmark-julia-packages-over-their-lifetime/97221/1 "2023-04-07T13:16:36Z")

</div>

# [AirspeedVelocity.jl](https://github.com/MilesCranmer/AirspeedVelocity.jl)

AirspeedVelocity.jl tries to make it easier to benchmark Julia packages over their lifetime.  
It is inspired by, and takes its name from, [asv](https://asv.readthedocs.io/en/stable/), (and aspires to one day have as nice a UI).

Basically, think of it as PkgBenchmarks.jl, but higher level. There are more built-in features, but it is also more rigid. AirspeedVelocity.jl started as a simple script I made to try to visualize the longterm performance evolution of my own packages but I thought it might be useful to others as well.

This package allows you to:

- Generate benchmarks directly from the terminal with an easy-to-use CLI (extremely handy when working side-by-side with git)
- Compare several commits/tags/branches at once.
- Plot generated benchmarks over time, with an automatically flattening of a hierarchical `BenchmarkGroup` suite into a list of plots with sensible subtitles.
- Includes an example GitHub action to generate benchmark comparisons for every submitted PR in a bot comment (table + plot).

This package also allows you to freeze a benchmark script at a particular revision, so there is no worry about using an older script. It also makes a `PACKAGE_VERSION` variable available for use in the benchmark, so you can switch to an older API within your script as needed.

## Installation

You can install the CLI with:

```bash
julia -e 'using Pkg; Pkg.add("AirspeedVelocity"); Pkg.build("AirspeedVelocity")'

```

This will install two executables at `~/.julia/bin`. Make sure to have it on your `PATH`.

## Examples

You may then use the CLI to generate benchmarks for any package that has a benchmark/benchmarks.jl` script:

```bash
benchpkg Transducers \
    --rev=v0.4.20,v0.4.70,master \
    --bench-on=v0.4.20

```

which will benchmark `Transducers.jl`, at the revisions `v0.4.20`, `v0.4.70`, and `master`, using the benchmark script `benchmark/benchmarks.jl` as it was defined at `v0.4.20`, and then save the JSON results in the current directory. The only requirement is that this script defines a `SUITE::BenchmarkGroup` (also used by `PkgBenchmark.jl`).

After this is finished, we can generate plots of the revisions with:

```bash
benchpkgplot Transducers \
    --rev=v0.4.20,v0.4.70,master \
    --format=pdf \
    --npart=5

```

which will generate a pdf file for each set of 5 plots,  
showing the change with each revision:

 ![Screenshot 2023-04-03 at 10 36 16 AM](https://global.discourse-cdn.com/julialang/original/3X/6/f/6fb626e24aaccfb3493e5a2c072b2c996d31463b.png)

There are a lot of other options - I’ll list those below. First, another feature I am excited about using for my own packages:

## Using in CI

You can use this package in GitHub actions to benchmark every submitted PR, by copying the example configuration: [`.github/workflows/benchmark_pr.yml`](https://github.com/MilesCranmer/AirspeedVelocity.jl/blob/master/.github/workflows/benchmark_pr.yml).

For every PR, or PR update, this workflow will run and generate plots of the performance of the PR against the default branch, as well as a markdown table (pasted into the comment), showing whether the PR improves or worsens performance:

 ![](https://global.discourse-cdn.com/julialang/original/3X/0/0/00f5268d178f8226e66fe813637852263612e88a.png)

## Usage

There are many other options for this CLI, which I give below. For running benchmarks, you can use the `benchpkg` command, which is built into the `~/.julia/bin` folder:

```plaintext
    benchpkg package_name [-r --rev <arg>] [-o, --output-dir <arg>]
                          [-s, --script <arg>] [-e, --exeflags <arg>]
                          [-a, --add <arg>] [--tune]
                          [--url <arg>] [--path <arg>]
                          [--bench-on <arg>]

Benchmark a package over a set of revisions.

# Arguments

- `package_name`: Name of the package.

# Options

- `-r, --rev <arg>`: Revisions to test (delimit by comma).
- `-o, --output-dir <arg>`: Where to save the JSON results.
- `-s, --script <arg>`: The benchmark script. Default: `benchmark/benchmarks.jl` downloaded from `stable`.
- `-e, --exeflags <arg>`: CLI flags for Julia (default: none).
- `-a, --add <arg>`: Extra packages needed (delimit by comma).
- `--url <arg>`: URL of the package.
- `--path <arg>`: Path of the package.
- `--bench-on <arg>`: If the script is not set, this specifies the revision at which
  to download `benchmark/benchmarks.jl` from the package.

# Flags

- `--tune`: Whether to run benchmarks with tuning (default: false).

```

This will generate some JSON files at the `output-dir` (default is current dir). For plotting, you can use the `benchpkgplot` function which will read in the same format:

```plaintext
    benchpkgplot package_name [-r --rev <arg>] [-i --input-dir <arg>]
                              [-o --output-dir <arg>] [-n --npart <arg>]
                              [--format <arg>]

Plot the benchmarks of a package as created with `benchpkg`.

# Arguments

- `package_name`: Name of the package.

# Options

- `-r, --rev <arg>`: Revisions to test (delimit by comma).
- `-i, --input-dir <arg>`: Where the JSON results were saved (default: ".").
- `-o, --output-dir <arg>`: Where to save the plots results (default: ".").
- `-n, --npart <arg>`: Max number of plots per page (default: 10).
- `--format <arg>`: File type to save the plots as (default: "png").

```

If you prefer to use the Julia REPL, you can use the `benchmark` function for generating data. The API is given [here](https://astroautomata.com/AirspeedVelocity.jl/dev/api/). (Although if you might just consider using PkgBenchmark.jl if wanting to customize things).

## Other notes

Non-stdlib dependencies include the following awesome packages:

- [Comonicon.jl](https://github.com/comonicon/Comonicon.jl) to build the CLI
- [PrettyTables.jl](https://github.com/ronisbr/PrettyTables.jl), for generating markdown tables in the GitHub actions comment
- [OrderedCollections.jl](https://github.com/JuliaCollections/OrderedCollections.jl), for assorted ordering tasks
- Plots.jl (GR) for the plots
- [FilePathsBase.jl](https://github.com/rofinn/FilePathsBase.jl) for some of the CLI relative-\>absolute path translation
- [JSON3.jl](https://github.com/quinnj/JSON3.jl) for serializing the benchmarks
- [BenchmarkTools.jl](https://github.com/JuliaCI/BenchmarkTools.jl) for actually defining and running the benchmarks

Thank you!

I am interested in hearing people’s thoughts and feedback. Package contributions are **very** welcome!  
Cheers,  
Miles

---

<div class="post-metadata">

**Author:** ![Krastanov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/krastanov/32/6817_2.png) [@Krastanov](https://discourse.julialang.org/u/Krastanov)\
**Post date:** [April 7, 2023, 1:27pm UTC](https://discourse.julialang.org/t/ann-airspeedvelocity-jl-easily-benchmark-julia-packages-over-their-lifetime/97221/2 "2023-04-07T13:27:45Z")

</div>

This is absolutely wonderful! Thank you so much for releasing it.

A ton of shameless feature requests/suggestions:

- How difficult would it be to also add a flag for custom julia version. E.g. thanks to juliaup I can do the following `julia +1.8.2 [other arguments]` and the Julia 1.8.2 executable will be used.
- What about TTL (time to load) measurements, i.e. benchmarking how long it takes to run `using MyPackage`?
- What about TTFX (time to first X, not including loading time) measurements, i.e. benchmarking `do_representative_workload()`?
- What about compilation time, i.e. measuring how long `] precompile` takes for an environment in which only the package is installed. I imagine this might be a bit more difficult as it requires deleting the entire cache. Maybe it can be done by pointing JULIA\_DEPOT to a temporary directory.
- Can we customize the number of threads for each of the above?
- Maybe using `time` to measure CPU load and peak memory usage?
- Storing the resulting data with a bunch of extra metadata, e.g. the output of `] st` and `versioninfo()`?

I have a variety of messy scripts made for personal use that do something like the above, but your package would be a much neater way to do all this.

---

<div class="post-metadata">

**Author:** ![MilesCranmer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/milescranmer/32/21070_2.png) [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)\
**Post date:** [April 7, 2023, 1:43pm UTC](https://discourse.julialang.org/t/ann-airspeedvelocity-jl-easily-benchmark-julia-packages-over-their-lifetime/97221/3 "2023-04-07T13:43:17Z")

</div>

I’m happy to hear that!! Not sure which ones could be done but I will think more. Responses below:

> [@Krastanov](#):
>
> - How difficult would it be to also add a flag for custom julia version. E.g. thanks to juliaup I can do the following `julia +1.8.2 [other arguments]` and the Julia 1.8.2 executable will be used.

Do you mean to have different Julia versions included as a matrix dimension in the benchmark? That could be interesting. It definitely is doable, as I’m just launching a separate Julia process entirely (I started with `addprocs`, but it became too complicated, so now I just start a new Julia for each benchmark).

> [@Krastanov](#):
>
> - What about TTL (time to load) measurements, i.e. benchmarking how long it takes to run `using MyPackage`?
> - What about TTFX (time to first X, not including loading time) measurements, i.e. benchmarking `do_representative_workload()`?
> - What about compilation time, i.e. measuring how long `] precompile` takes for an environment in which only the package is installed. I imagine this might be a bit more difficult as it requires deleting the entire cache. Maybe it can be done by pointing JULIA\_DEPOT to a temporary directory.

I think some of these might be doable by defining custom scripts with `-s` and a `BenchmarkGroup` for each? But indeed it might be nice to include TTL as a default measurement that gets concatenated with the user-defined BenchmarkGroup.

> [@Krastanov](#):
>
> - Can we customize the number of threads for each of the above?

You can pass `--exeflags` to the CLI, but it assumes there is only one per run.

Another option for all of these is to manually call `benchmark`: [API · AirspeedVelocity.jl](https://astroautomata.com/AirspeedVelocity.jl/dev/api/#Creating-benchmarks). You could loop over different options for the `exeflags`, and store it in an larger `BenchmarkGroup`.

> [@Krastanov](#):
>
> - Maybe using `time` to measure CPU load and peak memory usage?
> - Storing the resulting data with a bunch of extra metadata, e.g. the output of `] st` and `versioninfo()`?

These sound like good things to add. I’m not sure where to start though. Contributions welcome!

---

<div class="post-metadata">

**Author:** ![Krastanov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/krastanov/32/6817_2.png) [@Krastanov](https://discourse.julialang.org/u/Krastanov)\
**Post date:** [April 7, 2023, 1:52pm UTC](https://discourse.julialang.org/t/ann-airspeedvelocity-jl-easily-benchmark-julia-packages-over-their-lifetime/97221/4 "2023-04-07T13:52:09Z")

</div>

> [@MilesCranmer](#):
>
> Do you mean to have different Julia versions included as a matrix dimension in the benchmark?

Yes!

Also, yet another suggestion: If you benchmark an old version of some package, you might be pulling in a recent version of a dependency that significantly improved in performance (dependence on Polyester has led to that effect in my benchmarks). That is still a valid benchmark but it benchmarks “old package version on current julia ecosystem”. I frequently need a mode that is “old package version on a contemporary version of julia’s ecosystem”. This can be done by pulling locally an older version of the Registry repository.

---

<div class="post-metadata">

**Author:** ![MilesCranmer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/milescranmer/32/21070_2.png) [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)\
**Post date:** [June 18, 2024, 11:27am UTC](https://discourse.julialang.org/t/ann-airspeedvelocity-jl-easily-benchmark-julia-packages-over-their-lifetime/97221/5 "2024-06-18T11:27:09Z")

</div>

Many updates since I last posted… The nicest new feature is that the defaults are way smarter now. Basically all you need to do is run at the root of your package’s repo (in shell)

```bash
benchpkg

```

and it will:

1. Figure out the package name (from Project.toml)
2. Figure out the default branch name to compare the dirty state of your repo against
3. Evaluate all the benchmarks in `benchmarks/benchmark.jl` (BenchmarkTools.jl format – i.e., `const SUITE = BenchmarkGroup()`)
4. Print the result in a nicely formatted markdown table

This is nice to combine with the new `--filter` option. I like to quickly check if the load time has worsened compared to master, which I can do with

```bash
benchpkg --filter=time_to_load

```

Demo:

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [June 19, 2024, 4:08am UTC](https://discourse.julialang.org/t/ann-airspeedvelocity-jl-easily-benchmark-julia-packages-over-their-lifetime/97221/6 "2024-06-19T04:08:41Z")

</div>

> [@MilesCranmer](#):
>
> This will install two executables at `~/.julia/bin`. Make sure to have it on your `PATH`.

As someone who doesn’t have `~/.julia/bin` on my `PATH`, can I interest you in [Usage · BaseDirs.jl](https://tecosaur.github.io/BaseDirs.jl/stable/usage/#BaseDirs.User.bin)?

---

<div class="post-metadata">

**Author:** ![MilesCranmer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/milescranmer/32/21070_2.png) [@MilesCranmer](https://discourse.julialang.org/u/MilesCranmer)\
**Post date:** [June 19, 2024, 6:43am UTC](https://discourse.julialang.org/t/ann-airspeedvelocity-jl-easily-benchmark-julia-packages-over-their-lifetime/97221/7 "2024-06-19T06:43:37Z")

</div>

I guess this should be added to Comonicon.jl directly as I don’t modify the default behavior of the CLI generator. See [Create a CLI project · Comonicon.jl](https://comonicon.org/stable/project/#Setup-the-build.jl).

Looks like it’s set here: [Comonicon.jl/src/builder/install.jl at 43e0e9e307ae27b8965b0f8cb7696e8d57c697d7 · comonicon/Comonicon.jl · GitHub](https://github.com/comonicon/Comonicon.jl/blob/43e0e9e307ae27b8965b0f8cb7696e8d57c697d7/src/builder/install.jl#L32)

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [June 19, 2024, 6:59am UTC](https://discourse.julialang.org/t/ann-airspeedvelocity-jl-easily-benchmark-julia-packages-over-their-lifetime/97221/8 "2024-06-19T06:59:32Z")

</div>

Ah I see, I might open an issue there since I think it’s generally worth integrating with the host system when possible, and particularly when doing so lets the user skip extra steps like `PATH` modification.