[ANN] PerfChecker.jl - A small collection of packages performance checking and plotting tools

Hello there!

I’m glad to announce that I finally delivered my most “I have no idea what I am doing” based code in Julia as a brand-new package (should be released within half a day).

PerfChecker.jl is a collection of semi-automated performance checking tools for packages that compare a package through time (releases). Did I mention it output plots? Did I mention my skills with Plots.jl are terrible and that I am sure many people can help?

I have written a small tuto with a LOT of figures at GitHub - JuliaConstraints/PerfChecker.jl: A small collection of semi-automatic performance checking tools, and I would be delighted to have feedback. The tuto is based on the incoming v0.1.1 that will be released as soon as v0.1.0 is officially out (General Registry, I’m stalking you).

Just a couple of teasers:



link to repo ; seems like a fun project to me, nice work :+1:

(edit: small note that your docs link don’t work well on your repo readme)



The doc is currently empty (most of the content is available through the tuto), but I will write it down in the coming weeks after some feedback and fixes.

As for the link, I think the link to the stable version of the doc will work once the package is released (should happen soon, It has been 3 days since I registered it).


FYI, PerfChecker.jl is now at v0.1.1, so in a usable state! Any package developer should check it out.

I welcome help, as several aspects are currently out of my area of expertise. I am looking for wizards (or I have to become one, which is tempting but time-consuming)

For today, I wanted to share some of my goals regarding PerfChecker.jl and propose a tentative syntax for a v1. I would be delighted to have opinions and recommendations about it.

One goal of PerfChecker.jl is to have it execute performance checks in isolated environments (julia remote processes started with the relevant options). Ideally, I would like to have features similar to Test.jl.
However, unlike Test.jl, we would also need to execute perf checks in several versions of the package. Follows an attempt to propose a syntactic framework for such functionalities.

using PerfChecker

# Target(s)
using CompositionalNetworks

# Dependencies
using ConstraintDomains

# Start defining the performance checks
@checkset "Some title" begin

## Set compat ranges for Julia and targeted packages
## Each check will use the highest compatible version of Julia

# Define available julia versions and the respective executable paths
@juliaexe "1.3" "path/to/julia1.3.0.exe"
@juliaexe "1.6" "path/to/julia1.6.3.exe"

# Add target(s)
@target CompositionalNetworks "0.2" "0.3"
@notarget CompositionalNetworks "0.2.4"

# Some generic code used in the checks
domains = fill(domain([1, 2, 3]), 3)

## Triger compilation before checks. Note that code can be specified per version of the target(s)
f() = if version(CompositionalNetworks) ∈ v"0.2"
    foreach(_ -> explore_learn_compose(domains, allunique, somearg), 1:10)
    foreach((_ -> explore_learn_compose(domains, allunique), 1:10)

# Add compilation trigger for alloc check
@compile_alloc f

# Add compilation trigger for benchmark check (optional)
@compile_bench f

# Or add for both
@compile_check f

## Add checks
# alloc check
@alloc_check explore_learn_compose(domains, allunique)

# benchmark check (with optional keyargs similar to @benchmark)
@bench_check explore_learn_compose(domains, allunique) evals=1 samples=10 seconds=3600

## Plots would be generated based on csv files available and targets' versions ranges
@plots yaxis=:log10 kwarg2=option2 ...

end # @checkset

Such a perf-checker script would check allocations and benchmarks for each version of CompositionalNetworks. Each time, it would create an isolated environment by creating a new worker (and remove it after the check).

What do you think? Suggestions?

Interesting, I just thought about building something like that yesterday. Here are some features I am interested in, mostly for tracking progress for Makie:

  • Comparing not only registered versions but also arbitrary commits, PRs, branches, etc
  • Measuring the time it takes to do using Package
  • Maybe running each sample multiple times, to be more robust against natural performance deviations
  • Running the tests on CI and uploading results in some format that can be viewed or maybe interacted with online

Yes, I originally started with this feature in mind. I used registered versions in the first release, as it was the simplest to implement as a proof-of-concept.

Great idea. I bet there are some tricky parts to consider, though. Probably it could be worth to measure other things such as “download and precompile” from a clean environment, etc.

For allocation tracking and profiling, this could work in similar fashion than BenchmarkTools. Including the heuristic about the number of samples, evals, etc. Probably some tricky parts for that too haha.

Running the tests on CI would be good indeed. As for the last part, it sounds like a good job for Makie.jl!

I will update things here once I have something close to the syntax I proposed in my last post. From that point, it should be easier to include some of your ideas.

Thank you!

Personally, I would really try to avoid “macro soup”. It basically becomes a whole other language one has to learn, the error messages are usually not great etc. Some packages pull it off (like JuMP) but that is also a case where a DSL make sense. That’s my opinion at least.


That’s fair. I don’t have a strong opinion on the matter anyway.

As long as the user has a minimal list of actions to do for such a tool, I am fine with it.

Thanks for the feedback

Would also be great to run package benchmarks with a range of julia versions. Find potential performance regressions, and understand performance improvements: are they due to package update, or julia update?