# Regressions in Julia not getting caught by PkgEval

**URL:** <https://discourse.julialang.org/t/regressions-in-julia-not-getting-caught-by-pkgeval/121069>\
**Category:** General Usage\
**Tags:** pkgeval\
**Created:** [October 8, 2024, 7:29pm UTC](https://discourse.julialang.org/t/regressions-in-julia-not-getting-caught-by-pkgeval/121069 "2024-10-08T19:29:55Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [October 8, 2024, 7:29pm UTC](https://discourse.julialang.org/t/regressions-in-julia-not-getting-caught-by-pkgeval/121069/1 "2024-10-08T19:29:55Z")

</div>

As happened before as well, the 1.11 Julia release has some regressions that break existing code – and not just any code, but registered packages.  
Some examples of what I personally encountered today when testing my packages (that succeed on 1.10) on the new release: [Type of `@doc x` changes when REPL is loaded · Issue #54664 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/54664) (reported before), [1.11 regression: source file does not contain a module declaration · Issue #56050 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/56050), [in 1.11, `push!` doesn't forward to `append!` as it should · Issue #56051 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/56051).

These specific issues aren’t too difficult to work around in the code, that’s true. But shouldn’t PkgEval catch them before the release is made?  
Then either modify the change in Julia, or at least notify package authors.

Maybe I misunderstand the goals of PkgEval, and it’s not used to avoid package breakages in new Julia releases? Not sure, would be curious to hear about…

---

<div class="post-metadata">

**Author:** ![DanielVandH](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielvandh/32/31134_2.png) [@DanielVandH](https://discourse.julialang.org/u/DanielVandH)\
**Post date:** [October 8, 2024, 7:34pm UTC](https://discourse.julialang.org/t/regressions-in-julia-not-getting-caught-by-pkgeval/121069/2 "2024-10-08T19:34:43Z")

</div>

Here’s a report that did find that module declaration issue, [https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by\_date/2024-09/14/VOTables.primary.log](https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2024-09/14/VOTables.primary.log) with VOTables.jl. I guess once it’s in the " **2999 packages failed tests on the previous version too.**" category it gets skipped over…

Actually this report is on `Julia Version 1.12.0-DEV.1205` but I can’t find any reports on earlier versions.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [October 8, 2024, 7:37pm UTC](https://discourse.julialang.org/t/regressions-in-julia-not-getting-caught-by-pkgeval/121069/3 "2024-10-08T19:37:09Z")

</div>

> [@DanielVandH](#):
>
> failed tests on the previous version too

Interesting… Tests do pass for me (locally + CI) on 1.10.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [October 8, 2024, 9:42pm UTC](https://discourse.julialang.org/t/regressions-in-julia-not-getting-caught-by-pkgeval/121069/4 "2024-10-08T21:42:16Z")

</div>

“Previous version” here means the “comparison” commit it’s testing against. If you look at the PkgEval report, this is a bit clearer. Here’s the one from yesterday: [PkgEvalJob JuliaLang/julia@2ae0b7e [2024-10-06]](https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2024-10/06/report.html). There are packages whose tests are flaky and packages whose tests are plain old broken and packages whose tests were _already_ broken by something else and packages whose dependencies were broken.

Detangling all of the above takes significant time and energy. To put a finer point on it, PkgEval doesn’t catch it by itself. Someone _reads_ the PkgEval report and uses it to identify what went wrong (if anything) and then works to address it. You can look at all the pull requests in building the release candidates to see how times PkgEval was run and examined. It’s hard, but it is being done.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [October 9, 2024, 11:15am UTC](https://discourse.julialang.org/t/regressions-in-julia-not-getting-caught-by-pkgeval/121069/5 "2024-10-09T11:15:13Z")

</div>

I see, thank you both for the explanations!

Does it make sense to run PkgEval before the release or rc, comparing with the previous release – not with some nightly state? This event would be much more rare, enabling more effort (including community) to be put into reading and interpreting the report.

Note that the three linked issues (1st post) break tests of at least three different packages. By “break” I mean package or tests working on 1.10 and failing on 1.11.  
So it’s not some single package with randomly failing tests or internals usage.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [October 9, 2024, 11:39am UTC](https://discourse.julialang.org/t/regressions-in-julia-not-getting-caught-by-pkgeval/121069/6 "2024-10-09T11:39:24Z")

</div>

About this interesting case:

> [@aplavin](#):
>
> [in 1.11, `push!` doesn’t forward to `append!` as it should · Issue #56051](https://github.com/JuliaLang/julia/issues/56051)

And disagreement on that it “should”: [in 1.11, `push!` doesn't forward to `append!` as it should · Issue #56051 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/56051#issuecomment-2400509119)

Then I’m thinking, is there code that expects this (in packages, mostly or only?) and are they then considered buggy? Either way they break in 1.11? Meaning it seems like 1.11 broke semver? What is the cause of this (I suspect related to Memory type added), will the packages just be fixed and this likely never happens again for this, or can we expect similar for vector or other things about misaligned expectations?

And even if the “should” is not to be expected, then I agree PkgEval should catch, ideally… but it relies on tests, and they might always be incomplete.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [October 9, 2024, 11:52am UTC](https://discourse.julialang.org/t/regressions-in-julia-not-getting-caught-by-pkgeval/121069/7 "2024-10-09T11:52:20Z")

</div>

Whenever I read code A is equivalent to code B, I don’t assume A implements B or vice versa because it’s not possible for them to simultaneously implement each other in some equal relationship as “equivalent” implies. I just expect them to have the same results.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [October 9, 2024, 1:12pm UTC](https://discourse.julialang.org/t/regressions-in-julia-not-getting-caught-by-pkgeval/121069/8 "2024-10-09T13:12:00Z")

</div>

> [@aplavin](#):
>
> Does it make sense to run PkgEval before the release or rc, comparing with the previous release – not with some nightly state?

We often do so during the release process, see e.g. [Backports for 1.11.0-alpha2 · Pull Request #53543#issuecomment-1972874194 · JuliaLang/julia](https://github.com/JuliaLang/julia/pull/53543#issuecomment-1972874194). Subsequent runs that do not specify `vs=":release-1.10"` will compare against the merge-base, which is typically the previous alpha/beta/release candidate, and not the nightly.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [October 9, 2024, 1:38pm UTC](https://discourse.julialang.org/t/regressions-in-julia-not-getting-caught-by-pkgeval/121069/9 "2024-10-09T13:38:41Z")

</div>

Yep, and you can see that both V0Tables and Accessors were flagged as potentially having problems — and were in those ~1000 packages that [were manually rerun](https://github.com/JuliaLang/julia/pull/53543#issuecomment-1983748202) to try to detangle the above challenges.

This is also why RCs are announced and available. It’s all about trying to get more [Swiss cheese slices](https://en.wikipedia.org/wiki/Swiss_cheese_model) on the stack.