Integrating parallel vs single-threaded tests into Github CI

Tooling Package Management
1

Hi.

I’m not overly familiar with Github/CI and package management, and I’m working on a package that is performance-sensitive and likely to change a lot over time. Some changes are likely to affect performance, or multi-threaded code.

While I can easily find some packages to handle diagnosis (for memory allocations, timing, type stability issues, etc.), and while there are some approaches to benchmark tracking, I’m not sure how to handle ST vs MT code.

For instance, I’d like to set up a few tests to track significant performance changes over time, that multi-threaded code and single-threaded code are consistent In terms of results, and that MT code runs faster.

I’m not looking for extremely precise performance tests on the CI side ; I just want to make sure a performance baseline is met, have a few consistency checks, and make sure that no egregious changes are introduced.

I’m not sure how I should handle future regression analysis on the CI side, given that the number of threads is provided at startup. In C/C++, I could easily set up that kind of test because I can launch threads manually and not have to deal with some awkward setup on the CI side.

Are there articles or repositories showcasing what I want to do ? The only relevant thread I could find is 6 years old : Julia Multithreading (Threads.@threads) and Travis CI

2

I don’t think comparison between single- and multi-threaded benchmarks will be easy to visualize. But multi-threaded benchmarks on their own are possible (although the GitHub actions runners only have 2 or 4 threads IIRC).
You can find an example here where I used PkgBenchmark.jl and BenchmarkCI.jl for multithreaded benchmarks. Although since the latter is deprecated, I would recommend AirspeedVelocity.jl these days. With that package, you can pass custom command line flags to Julia, see the README for details.