Codecov is a very useful tool to assess code coverage, that is, how much of the package source code is hit by the test suite. In GitHub continuous integration workflows, this happens through the codecov-action, which recently released a breaking version 4. As a result, many package maintainers are receiving pull requests from Dependabot to update their continuous integration CI.yml files., with changes like the following:
If you just merge this without other changes, your coverage stats will no longer update and the README badge will show a wrong percentage. Indeed, uploading these stats to the Codecov web service now requires setting a repository or organization secret. As stated in the release notes:
Tokenless uploading is unsupported. However, PRs made from forks to the upstream public repos will support tokenless (e.g. contributors to OS projects do not need the upstream repo’s Codecov token). This doc shows instructions on how to add the Codecov token.
What you need to do
If you are a package maintainer, read the Codecov documentation on tokens and follow the instructions there. For each individual repository, you need to:
Retrieve a token from the Codecov settings of that repository
Add this token as a secret in the GitHub settings of that repository. In the secret value box, don’t put CODECOV_TOKEN = abcd1234, just put abcd1234.
Update the testing part of the CI.yml file like so:
- uses: codecov/codecov-action@v4
with:
# possibly other stuff
token: ${{ secrets.CODECOV_TOKEN }}
fail_ci_if_error: false # or true if you want CI to fail when Codecov fails
If you are an organization owner, part of the work can be mutualized:
The secret can be set in the organization settings on GitHub
Each package’s CI.yml still needs an individual update [sigh]
Possible issues:
The setting fail_ci_if_error: true might lead to unexpected CI errors that are due to Codecov server status or other things beyond our control.
Lessons learned
When you receive a Dependabot PR, check the release notes for breaking changes like this one. They are usually in a collapsed section of the PR. I know I didn’t use to, but now I will.
Run codecov/codecov-action@v4
==> linux OS detected
https://cli.codecov.io/latest/linux/codecov.SHA256SUM
==> Running version latest
==> Running version v0.4.6
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-commit
gpg: directory '/home/runner/.gnupg' created
gpg: keybox '/home/runner/.gnupg/pubring.kbx' created
gpg: /home/runner/.gnupg/trustdb.gpg: trustdb created
gpg: key 806BB28AED779869: public key "Codecov Uploader (Codecov Uploader Verification Key) <security@codecov.io>" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: Signature made Fri Feb 2 14:15:33 2024 UTC
gpg: using RSA key 27034E7FDB850E0BBC2C62FF806BB28AED779869
gpg: Good signature from "Codecov Uploader (Codecov Uploader Verification Key) <security@codecov.io>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 2703 4E7F DB85 0E0B BC2C 62FF 806B B28A ED77 9869
==> Uploader SHASUM verified (103bfefcc56f76473179e600b96eb8150b0f349ad94836b0f63f03ffac469ad7 codecov)
info - 2024-02-07 13:18:34,767 -- ci service found: github-actions
warning - 2024-02-07 13:18:34,770 -- No config file could be found. Ignoring config.
info - 2024-02-07 13:18:34,997 -- Process Commit creating complete
error - 2024-02-07 13:18:34,998 -- Commit creating failed: {"error": "Server Error (500)"}
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-report'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov create-report
info - 2024-02-07 13:18:35,704 -- ci service found: github-actions
warning - 2024-02-07 13:18:35,707 -- No config file could be found. Ignoring config.
info - 2024-02-07 13:18:35,943 -- Process Report creating complete
error - 2024-02-07 13:18:35,944 -- Report creating failed: {"error": "Server Error (500)"}
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov do-upload'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov do-upload -f lcov.info
info - 2024-02-07 13:18:36,648 -- ci service found: github-actions
warning - 2024-02-07 13:18:36,651 -- No config file could be found. Ignoring config.
warning - 2024-02-07 13:18:36,658 -- xcrun is not installed or can't be found.
warning - 2024-02-07 13:18:36,660 -- No gcov data found.
warning - 2024-02-07 13:18:36,660 -- coverage.py is not installed or can't be found.
info - 2024-02-07 13:18:36,672 -- Found 1 coverage files to upload
info - 2024-02-07 13:18:36,672 -- > /home/runner/work/ABCDMatrixOptics.jl/ABCDMatrixOptics.jl/lcov.info
info - 2024-02-07 13:18:36,877 -- Process Upload complete
error - 2024-02-07 13:18:36,878 -- Upload failed: {"error": "Server Error (500)"}
EDIT: here the same config works. So it might be related to the JuliaPhysics org.
JuliaDSP is using it successfully. But I have heard other orgs are having some challenges. Are there some settings that need toggling, like some permissions?
The codecov website was also a bit unstable yesterday, perhaps because everyone is doing the same thing. I suppose it will settle down in a few days.
Wrt to the original announcement and subsequent org-level tokens being added, it’s also worth noting that the Codecov action v3 can’t use org-level tokens. You should therefore update your CI file along with or after the Codecov v4 update. If you change your CI file to use the org-level token while still using Codecov v3, you would encounter errors uploading the coverage reports.
I’m getting “Server error (500)” on all of my personal and organization repos, including Graphs.jl, both yesterday and today. This is regardless of the tests passing, cause I set fail_ci_if_error: false: see the CI log.
I’d appreciate any help figuring this out! I hope this is temporary but the status page of Codecov.io is all green.
EDIT: it was because I stored the secret in the wrong way. In the GitHub box for the secret value, put only abcd1234, not CODECOV_TOKEN=abcd1234. I edited the original post
