PerfChecker.jl is reaching a good state and we're looking for ~~labrats~~ testers/users

[Azzaare]

Hi there!

Years ago, I announced a proof of concept package to handle testing performances over versions: PerfChecker.jl

We’re reaching the point were the package is ready to be used (v"0.2.0" waiting for merge). And we would be very grateful to have some testers/users to have some feedbacks before JuliaCon. There is a lot of room for improvements, and we haven’t yet integrated all the feedbacks from last JuliaCon poster !

EDIT: I am working on the docs, so please refer to the scripts below in the meantime.

As the features evolved a bit over time, allow me to list some of them here:

• Built-in profiling over versions
• Generic Interface for extensions
  • BenchmarkTools extension (benchmarks)
  • Chairmarks extension (benchmarks)
  • Makie extension (plotting)
• Sugar syntax with macro
• Checks run in isolated environments using Malt.jl’s processes
• Tags system to load/save results
• UUIDs for machines and checks setup to avoid redundant computations (this is an effort towards CI integration). It is the first step to save metadata.

We provide basic scripts to check performances for only 2 packages so far: GLM.jl and PatternFolds.jl. This is where helps to provide scripts for other packages. Hey, with a bit of luck, you might find out some performances issues! Or you can just brag how much your last version is better than before.

Let me show you below a few scripts example for PatternFolds.jl. For each script, we also generate plots through CairoMakie.

using PerfChecker, CairoMakie

d = Dict(:targets => ["PatternFolds"], :path => @__DIR__, :tags => [:patterns, :intervals],
    :pkgs => (
        "PatternFolds", :custom, [v"0.2.0", v"0.2.1", v"0.2.2", v"0.2.3", v"0.2.4"], true))

x = @check :alloc d begin
    # Pre check script
    using PatternFolds
end begin
    # check script
    itv = Interval{Open, Closed}(0.0, 1.0)
    i = IntervalsFold(itv, 2.0, 1000)

    unfold(i)
    collect(i)
    reverse(collect(i))

    # Vectors
    vf = make_vector_fold([0, 1], 2, 1000)

    unfold(vf)
    collect(vf)
    reverse(collect(vf))

    rand(vf, 1000)
end

@info x

mkpath(joinpath(@__DIR__, "visuals"))

c = checkres_to_scatterlines(x, Val(:alloc))
save(joinpath(@__DIR__, "visuals", "allocs_evolution.png"), c)

for (name, c2) in checkres_to_pie(x, Val(:alloc))
    save(joinpath(@__DIR__, "visuals", "allocs_pie_$name.png"), c2)
end

We can clearly see the drop in allocations between version 0.2.1 and 0.2.2.

If we look at benchmarks (with the following script):

using PerfChecker, Chairmarks, CairoMakie

d = Dict(:path => @__DIR__, :evals => 10, :samples => 1000,
    :seconds => 100, :tags => [:patterns, :intervals],
    :pkgs => (
        "PatternFolds", :custom, [v"0.2.0", v"0.2.1", v"0.2.2", v"0.2.3", v"0.2.4"], true),
    :devops => "PatternFolds")

x = @check :chairmark d begin
    using PatternFolds
end begin
    # Intervals
    itv = Interval{Open, Closed}(0.0, 1.0)
    i = IntervalsFold(itv, 2.0, 1000)

    unfold(i)
    collect(i)
    reverse(collect(i))

    # Vectors
    vf = make_vector_fold([0, 1], 2, 1000)

    unfold(vf)
    collect(vf)
    reverse(collect(vf))

    rand(vf, 1000)

    return nothing
end

@info x

mkpath(joinpath(@__DIR__, "visuals"))

c = checkres_to_scatterlines(x, Val(:chairmark))
save(joinpath(@__DIR__, "visuals", "chair_evolution.png"), c)

for kwarg in [:times, :gctimes, :bytes, :allocs]
    c2 = checkres_to_boxplots(x, Val(:chairmark); kwarg)
    save(joinpath(@__DIR__, "visuals", "chair_boxplots_$kwarg.png"), c2)
end

We can see that a small increase in speed (likely due to the drop in allocations, … might need log scale …)

We would be very happy with testing our tool on other packages (we do have some machines with some free resources here and there). Or have you test them directly. Scripts (and even output) can be added in PerfChecker.jl/perf through PRs.

[Azzaare]

Hi there!

About two years after my last post, I’m back asking for labrats testers PerfChecker.jl v1.0.0-rc1 is ready to try.

It took a bit longer than expected.

The idea is still the same: run the same code over several versions of a package and see how its performance evolves. Did an update make something slower? Did reducing allocations actually help? At which release did things change?

A lot has changed around this since v0.2, so here is a small tour.

Comparing versions, then looking into the changes

The measurements and analysis tools have grown quite a bit since the previous version. BenchmarkTools and Chairmarks were already there, but there is now more to investigate what happens around a timing change:

• Benchmarks: execution time, allocated bytes, allocation count, and GC time or GC fraction, depending on the backend. You can compare distributions across versions, plot each measure separately, or put the normalized curves together.
• CPU profiles: sampled call stacks to see where execution time goes.
• Wall-time profiles: task stacks, including waiting tasks. Useful when the program is slow but the CPU profile doesn’t explain why. This needs Julia 1.12 or later.
• Allocation profiles: allocation sites, object types, sampled bytes and event counts through Profile.Allocs, plus allocation tracking by source file and line.
• Heap snapshots: inspect the objects still in Julia’s managed heap, with summaries by type and shallow size. This answers a different question from “how much did this operation allocate?”
• Process memory: resident memory, private memory and process lifetime peaks on Windows and Linux. These also help spot memory growth that Julia allocation counters don’t explain.
• External memory: callbacks for packages which manage native buffers, pools or other memory outside Julia’s heap. They can report live, reserved, allocated and freed bytes, and check that memory is released as expected.
• Garbage collection: collection time, collection counts and allocation pressure around an operation.
• Loading and first-call latency: separate source loading, the first operation and warm execution. SnoopCompile is also available to investigate inference work.
• Lock contention: observed ReentrantLock conflicts, to help investigate workloads that don’t improve when adding threads.
• Network traffic: sent and received bytes and packets, alongside workload latency and throughput. There are application-reported counters, host-interface counters, and isolated process-tree measurements through Linux network namespaces, including through WSL2 on Windows.

There are also checks which don’t measure speed directly, but can help explain a problem:

• JET for inference and dynamic dispatch diagnostics.
• AllocCheck for possible allocations in compiled calls.
• Aqua for package quality checks.
• CoverageTools to relate workloads to the source they exercise.
• PropCheck and Supposition for generated test cases and reproducible counterexamples.

For parallel and GPU work, there are shared scenarios using Threads, Distributed, local Dagger tasks, and KernelAbstractions on CPU or CUDA. These make preparation, synchronization and correctness checks explicit. They are examples to build on; GPU profiler support still depends on the device and tools available.

I’ve also added documentation and command preparation for native tools: Valgrind’s Memcheck, Callgrind, Cachegrind and Massif, heaptrack, perf and VTune. This matters when the expensive part, or the memory problem, is inside a C/C++/Fortran dependency. These require separate installation; they are not all integrated collectors.

And for reading the results, there are interactive flame graphs, allocation breakdowns, heatmaps, version comparisons and distributions, with Makie figures, PProf exports and UnicodePlots in the terminal. DrWatson integration is available for keeping experiments and their results organized.

It’s quite a long list now The examples show how these tools can be used on the same workloads, so you don’t have to start by figuring out how to connect everything yourself.

Using tests you already have

One addition I wanted is the possibility to reuse existing @testitems. You can select individual items or filter them by tags, without maintaining a completely separate set of examples for performance checking.

There is a distinction here: measuring a whole test includes its setup, assertions and cleanup. If you want to benchmark one operation precisely, you can define that separately and keep the preparation outside the measurement.

Both are useful, depending on what you want to check.

Several ways to use it

There are now interfaces for:

• VS Code, to discover and run individual items while working on your package.
• The browser, through an Oxygen-based web interface.
• Pluto, with notebooks you can edit and reuse.
• The REPL and scripts, including workflows for CI.

They use the same saved results. You can run measurements from a script and explore them later in another interface.

The interface packages are optional. The VS Code extension is also still an early version, distributed separately for now.

Examples you can actually start from

I’ve spent quite some time on the documentation. There is a quickstart, but also longer examples with real packages:

• DataStructures.jl: separate workloads for different data structures, compared across historical releases.
• Oxygen.jl: different HTTP features, including measurements with actual network traffic.

These include plots, profiling results, scripts and notebooks to adapt for your own packages. The measurements are already available, so you don’t need to run the full campaign just to explore the results.

I’d also be happy to include examples from other people, especially if PerfChecker helped you find a regression or check an improvement.

Trying the release candidate

The release candidate has a different API from v0.2. To try it in a separate Julia environment:

import Pkg
Pkg.add(Pkg.PackageSpec(
    name = "PerfChecker",
    rev = "release/v1.0.0-rc1",
))

The optional interface packages have their own installation instructions in the docs; they are not all registered yet.

Before tagging v1.0, I’d really appreciate feedback from people using it on their own packages. In particular: can you get a first result without fighting the setup? Are the comparisons useful? Do the docs explain enough to reproduce the examples?

And of course, bugs, missing features, or “why do I need three steps to do this?” are welcome too. There is still time to fix awkward parts before v1.

Documentation · GitHub and issues

Thanks!