Coverage and precompilation

General Usage
1

Hi!

When I add precompilation statements from SnoopCompile, I start to see weird coverage results. Take a look here:

Captura de Tela 2022-03-02 às 19.35.59
Captura de Tela 2022-03-02 às 19.35.591468×620 80.5 KB

Everything seems fine when I remove those statements.

I saw that this can be improved if I run Julia with --inline=no when generating the coverage information. However, I am using the Github action julia-actions/julia-processcoverage@v1. Can anyone tell me what can I do to add this --inline=no option to that Github action?

2

That action actually just collects coverage files already generated by previous steps in your workflow, most likely from julia-actions/julia-runtest. To add --inline=no there, you change

      - uses: julia-actions/julia-runtest@v1

to

      - uses: julia-actions/julia-runtest@v1
        with:
          inline: no

Note: some code heavily depends on inlining for performance. I saw a test go from a few seconds to 300s+ with --inline=no! So you might want to be careful where you use it.

2 Likes
3

Thanks @ericphanson !

It does not have any time impact. However, the coverage did not improve :frowning: I am still seeing things like this if I use precompilation directives:

Captura de Tela 2022-03-03 às 10.08.26
Captura de Tela 2022-03-03 às 10.08.26946×224 21 KB

4

I am wondering if I can remove the precompilation if I am running for coverage computation.

5

It is ugly, but seems to work:

First, I add the precompilation directive as follows:

if Base.VERSION >= v"1.4.2" && !haskey(ENV, "STRING_MANIPULATION_NO_PRECOMPILATION")
    # This try/catch is necessary in case the precompilation statements do not
    # exists. In this case, StringManipulation.jl will work correctly but
    # without the optimizations.
    try
        include("../precompilation/precompile_StringManipulation.jl")
        _precompile_()
    catch
    end
end

Then, I removed the coverage processing from the main GitHub action.

After that, I create another action for the coverage using the following file:

name: Coverage
on:
  pull_request:
    branches:
      - main
  push:
    branches:
      - main
    tags: '*'
jobs:
  test:
    name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        version:
          - '1'  # This automatically expands to the latest stable 1.x release.
        os:
          - ubuntu-latest
        arch:
          - x64
    env:
      STRING_MANIPULATION_NO_PRECOMPILATION: ""
    steps:
      - uses: actions/checkout@v2
      - uses: julia-actions/setup-julia@v1
        with:
          version: ${{ matrix.version }}
          arch: ${{ matrix.arch }}
      - uses: actions/cache@v1
        env:
          cache-name: cache-artifacts
        with:
          path: ~/.julia/artifacts
          key: ${{ runner.os }}-test-${{ env.cache-name }}-${{ hashFiles('**/Project.toml') }}
          restore-keys: |
            ${{ runner.os }}-test-${{ env.cache-name }}-
            ${{ runner.os }}-test-
            ${{ runner.os }}-
      - uses: julia-actions/julia-buildpkg@v1
      - uses: julia-actions/julia-runtest@v1
      - uses: julia-actions/julia-processcoverage@v1
      - uses: codecov/codecov-action@v2
        with:
          file: lcov.info

And now everything works :slight_smile:

Captura de Tela 2022-03-03 às 10.34.33
Captura de Tela 2022-03-03 às 10.34.331888×1304 310 KB

3 Likes
6

It would be interesting to know if this also happens on nightly, which has major changes to precompilation.

Off the top of my head, the most likely culprit may be constant-propagation, and if different code gets generated depending on whether you precompile.

2 Likes
7

Hi @tim.holy !

Thanks for the information, I will check :slight_smile: