# CI : benchmarking dependencies best practices?

**URL:** <https://discourse.julialang.org/t/ci-benchmarking-dependencies-best-practices/128402>\
**Category:** Package Management\
**Tags:** benchmark, ci, dependencies\
**Created:** [April 25, 2025, 1:36pm UTC](https://discourse.julialang.org/t/ci-benchmarking-dependencies-best-practices/128402 "2025-04-25T13:36:10Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![s\_amap](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@s\_amap](https://discourse.julialang.org/u/s_amap)\
**Post date:** [April 25, 2025, 1:36pm UTC](https://discourse.julialang.org/t/ci-benchmarking-dependencies-best-practices/128402/1 "2025-04-25T13:36:10Z")

</div>

I’m working on several packages, with one or two packages with core features, and others depending on them, which aren’t really develop-centric, more so user applications.

When adding features to the core packages, there are integration checks to ensure that no breaking changes were accidentally introduced, meaning that we can create a PR, update our downstream packages smoothly, and ensure things are kept in sync as quickly as possible.

This works fine, but there is one area I haven’t been able to decide on in terms of how I should architect CI : benchmarking.

I’ve no issues setting up non-regression benchmarks for a single package, or ad hoc benchmarks for more specialised throwaway tests, but when dependencies get involved, things get a little finickier.

Example : one of our dependencies had a performance issue which was fixed in one of the core package. That’s a PR on the core package side, but a slight API change requires updating versions of dependencies, meaning it’s an issue that requires updating both packages, so you can’t just update the core package and simply check before/after on that end only. It is not trivial to set up a core package-exclusive equivalent showcasing the same performance issue, so both packages are necessary.

Seems to me non-regression benchmarks make more sense to be run in a centralised manner on the core packages where more dev activity happens. But I also want to pointlessly avoid running them when CI isn’t up-to-date in terms of integration, and I don’t want to have regressions bleed out into releases.

One way of keeping things clean would be to have a specific downstream benchmark test in a core package which pulls in dependencies in their ‘pre-release’ branch, after integration checks have been validated. That sounds like it fits my needs but it also feels clunky to have a main branch and a pre-release branch.

Or should I just benchmark on the dependency end ? How do other packages deal with this smoothly ?

I get the impression some Julia benchmarking tools seem to make plotting performance changes over time to detect regressions possible, but not that straightforward to combine with dependencies.

---

<div class="post-metadata">

**Author:** ![s\_amap](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@s\_amap](https://discourse.julialang.org/u/s_amap)\
**Post date:** [April 25, 2025, 1:39pm UTC](https://discourse.julialang.org/t/ci-benchmarking-dependencies-best-practices/128402/2 "2025-04-25T13:39:48Z")

</div>

I guess there are multiple interrelated aspects to this question :

- Non-regression benchmarking dependencies isn’t discussed at much on the Github/CI side
- Whether to centralize benchmarks of multiple packages or not, and how convenient that might be
- Ensuring sync for benchmarking dependencies

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [April 26, 2025, 7:21am UTC](https://discourse.julialang.org/t/ci-benchmarking-dependencies-best-practices/128402/3 "2025-04-26T07:21:31Z")

</div>

@willow maybe?

---

<div class="post-metadata">

**Author:** ![willow](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/willow/32/208918_2.png) [@willow](https://discourse.julialang.org/u/willow)\
**Post date:** [May 8, 2025, 12:12am UTC](https://discourse.julialang.org/t/ci-benchmarking-dependencies-best-practices/128402/4 "2025-05-08T00:12:12Z")

</div>

I understand the desire to have a downstream benchmark for upstream packages. Could you add a benchmark for the downstream package as one of the benchmarks for the core package? That would at least provide feedback in the same place you have power to change it.