For today, I wanted to share some of my goals regarding PerfChecker.jl and propose a tentative syntax for a
v1. I would be delighted to have opinions and recommendations about it.
One goal of PerfChecker.jl is to have it execute performance checks in isolated environments (julia remote processes started with the relevant options). Ideally, I would like to have features similar to Test.jl.
However, unlike Test.jl, we would also need to execute perf checks in several versions of the package. Follows an attempt to propose a syntactic framework for such functionalities.
# Start defining the performance checks
@checkset "Some title" begin
## Set compat ranges for Julia and targeted packages
## Each check will use the highest compatible version of Julia
# Define available julia versions and the respective executable paths
@juliaexe "1.3" "path/to/julia1.3.0.exe"
@juliaexe "1.6" "path/to/julia1.6.3.exe"
# Add target(s)
@target CompositionalNetworks "0.2" "0.3"
@notarget CompositionalNetworks "0.2.4"
# Some generic code used in the checks
domains = fill(domain([1, 2, 3]), 3)
## Triger compilation before checks. Note that code can be specified per version of the target(s)
f() = if version(CompositionalNetworks) ∈ v"0.2"
foreach(_ -> explore_learn_compose(domains, allunique, somearg), 1:10)
foreach((_ -> explore_learn_compose(domains, allunique), 1:10)
# Add compilation trigger for alloc check
# Add compilation trigger for benchmark check (optional)
# Or add for both
## Add checks
# alloc check
@alloc_check explore_learn_compose(domains, allunique)
# benchmark check (with optional keyargs similar to @benchmark)
@bench_check explore_learn_compose(domains, allunique) evals=1 samples=10 seconds=3600
## Plots would be generated based on csv files available and targets' versions ranges
@plots yaxis=:log10 kwarg2=option2 ...
end # @checkset
Such a perf-checker script would check allocations and benchmarks for each version of CompositionalNetworks. Each time, it would create an isolated environment by creating a new worker (and remove it after the check).
What do you think? Suggestions?