PSA: new version of codecov-action requires additional setup

What’s going on

Codecov is a very useful tool to assess code coverage, that is, how much of the package source code is hit by the test suite. In GitHub continuous integration workflows, this happens through the codecov-action, which recently released a breaking version 4. As a result, many package maintainers are receiving pull requests from Dependabot to update their continuous integration CI.yml files., with changes like the following:

codecov/codecov-action@v3 -> codecov/codecov-action@v4

If you just merge this without other changes, your coverage stats will no longer update and the README badge will show a wrong percentage. Indeed, uploading these stats to the Codecov web service now requires setting a repository or organization secret. As stated in the release notes:

Tokenless uploading is unsupported. However, PRs made from forks to the upstream public repos will support tokenless (e.g. contributors to OS projects do not need the upstream repo’s Codecov token). This doc shows instructions on how to add the Codecov token.

What you need to do

If you are a package maintainer, read the Codecov documentation on tokens and follow the instructions there. For each individual repository, you need to:

  1. Retrieve a token from the Codecov settings of that repository
  2. Add this token as a secret in the GitHub settings of that repository. In the secret value box, don’t put CODECOV_TOKEN = abcd1234, just put abcd1234.
  3. Update the testing part of the CI.yml file like so:
- uses: codecov/codecov-action@v4
    with:
      # possibly other stuff
      token: ${{ secrets.CODECOV_TOKEN }}
      fail_ci_if_error: false  # or true if you want CI to fail when Codecov fails

If you are an organization owner, part of the work can be mutualized:

  1. The token can be retrieved in the organization settings on Codecov, using a link like https://app.codecov.io/account/gh/MyJuliaOrg
  2. The secret can be set in the organization settings on GitHub
  3. Each package’s CI.yml still needs an individual update [sigh]

Possible issues:

  • The setting fail_ci_if_error: true might lead to unexpected CI errors that are due to Codecov server status or other things beyond our control.

Lessons learned

When you receive a Dependabot PR, check the release notes for breaking changes like this one. They are usually in a collapsed section of the PR. I know I didn’t use to, but now I will.

31 Likes

Thank you @gdalle for sharing these instructions. Really helpful :heart:

2 Likes

I tried unsuccessfully so far. Can anyone share a working repository?

In my case I get some upload errors now:

log
Run codecov/codecov-action@v4
==> linux OS detected
https://cli.codecov.io/latest/linux/codecov.SHA256SUM
==> Running version latest
==> Running version v0.4.6
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit
gpg: directory '/home/runner/.gnupg' created
gpg: keybox '/home/runner/.gnupg/pubring.kbx' created
gpg: /home/runner/.gnupg/trustdb.gpg: trustdb created
gpg: key 806BB28AED779869: public key "Codecov Uploader (Codecov Uploader Verification Key) <security@codecov.io>" imported
gpg: Total number processed: 1
gpg:               imported: 1

gpg: Signature made Fri Feb  2 14:15:33 2024 UTC
gpg:                using RSA key 27034E7FDB850E0BBC2C62FF806BB28AED779869
gpg: Good signature from "Codecov Uploader (Codecov Uploader Verification Key) <security@codecov.io>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 2703 4E7F DB85 0E0B BC2C  62FF 806B B28A ED77 9869

==> Uploader SHASUM verified (103bfefcc56f76473179e600b96eb8150b0f349ad94836b0f63f03ffac469ad7  codecov)
info - 2024-02-07 13:18:34,767 -- ci service found: github-actions
warning - 2024-02-07 13:18:34,770 -- No config file could be found. Ignoring config.
info - 2024-02-07 13:18:34,997 -- Process Commit creating complete
error - 2024-02-07 13:18:34,998 -- Commit creating failed: {"error": "Server Error (500)"}
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-report'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-report
info - 2024-02-07 13:18:35,704 -- ci service found: github-actions
warning - 2024-02-07 13:18:35,707 -- No config file could be found. Ignoring config.
info - 2024-02-07 13:18:35,943 -- Process Report creating complete
error - 2024-02-07 13:18:35,944 -- Report creating failed: {"error": "Server Error (500)"}
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov do-upload'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov do-upload -f lcov.info
info - 2024-02-07 13:18:36,648 -- ci service found: github-actions
warning - 2024-02-07 13:18:36,651 -- No config file could be found. Ignoring config.
warning - 2024-02-07 13:18:36,658 -- xcrun is not installed or can't be found.
warning - 2024-02-07 13:18:36,660 -- No gcov data found.
warning - 2024-02-07 13:18:36,660 -- coverage.py is not installed or can't be found.
info - 2024-02-07 13:18:36,672 -- Found 1 coverage files to upload
info - 2024-02-07 13:18:36,672 -- > /home/runner/work/ABCDMatrixOptics.jl/ABCDMatrixOptics.jl/lcov.info
info - 2024-02-07 13:18:36,877 -- Process Upload complete
error - 2024-02-07 13:18:36,878 -- Upload failed: {"error": "Server Error (500)"}

EDIT: here the same config works. So it might be related to the JuliaPhysics org.

JuliaDSP is using it successfully. But I have heard other orgs are having some challenges. Are there some settings that need toggling, like some permissions?

The codecov website was also a bit unstable yesterday, perhaps because everyone is doing the same thing. I suppose it will settle down in a few days.

-viral

1 Like

I can only speak for JuliaPhysics where all my repos fail currently. JuliaGraphs fails too according to @gdalle.

Maybe @giordano knows more?

Yeah, adding to token for JuliaPhysics is in my TODO list.

But if I use the private key for my repo? Shouldn’t that work?

In case it affected someone, there was a typo (bracket issue) in the CI excerpt above, fixed now

I set up the org-wide key in JuliaPhysics, and upload is working in [GHA] Switch to `codecov/codecov-action` by giordano · Pull Request #141 · JuliaPhysics/Measurements.jl · GitHub

I’m still confused why a separate key would not work?

But how could I access the key?

It’s working now after deleting the key you created, and automatically using the org-wide one: Update CI.yml · JuliaPhysics/ABCDMatrixOptics.jl@0d678bf · GitHub. Sounds like it was wrong?

2 Likes

Wrt to the original announcement and subsequent org-level tokens being added, it’s also worth noting that the Codecov action v3 can’t use org-level tokens. You should therefore update your CI file along with or after the Codecov v4 update. If you change your CI file to use the org-level token while still using Codecov v3, you would encounter errors uploading the coverage reports.

Repo tokens work with either version.

1 Like

I’m getting “Server error (500)” on all of my personal and organization repos, including Graphs.jl, both yesterday and today. This is regardless of the tests passing, cause I set fail_ci_if_error: false: see the CI log.
I’d appreciate any help figuring this out! I hope this is temporary but the status page of Codecov.io is all green.

EDIT: it was because I stored the secret in the wrong way. In the GitHub box for the secret value, put only abcd1234, not CODECOV_TOKEN=abcd1234. I edited the original post

3 Likes