Testing a package extension?

General Usage
,
1

General question is: what is the recommended way to run CI tests on functionality provided in package extensions?

One option would be I guess to make the tests require all the possible extension-triggering weak-dependencies. I do not want to do this, because I want users to be able to run tests without installing dependencies that they do not want (I’m happy to add some code to skip tests if the extension is not loaded - if I have trouble with that, it’s a separate issue/question…).

Suppose I have a package which has an extension to provide some extra functionality (e.g. output to NetCDF if NCDatasets is loaded). I run CI tests with a workflow like

name: Run tests

on: [push, pull_request, workflow_dispatch]

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, macOS-latest]
      fail-fast: false
    timeout-minutes: 50

    steps:
      - uses: actions/checkout@v4
      - uses: julia-actions/setup-julia@v1
        with:
          version: '1.8'
          arch: x64
      - uses: julia-actions/julia-buildpkg@v1
      - uses: julia-actions/julia-runtest@v1

What would be a good way to get the extension-triggering weak-dependencies loaded? I tried modifying the workflow to install NCDatasets into the project, but that seems to fail (i.e. the extension is not loaded), I guess because NCDatasets is already in the [weakdeps] section? [I’m testing this with julia-1.9.4, in case the version is relevant.]

It seems like maybe the only thing to do is to create a separate project environment, dev my package into that environment, add the extension-triggering weak-dependency too, and then run the tests there? I guess that is doable, but more complicated than I would like for something that seems like any package using extensions probably wants to do?

2

My current thought for a workaround is:

  • add NCDatasets to the test dependencies, so it will be installed when using Pkg.test().
  • for users who want to run the tests without extra dependencies, they can run test/runtests.jl directly, which (I think?) does not know about the ‘test dependencies’ from the Project.toml.
3

Another workaround would be to add the triggering dependencies in the CI script before the tests are run, but that would be rather dirty since Pkg also does env manipulations for testing

4

Another workaround would be to add the triggering dependencies in the CI script before the tests are run, but that would be rather dirty since Pkg also does env manipulations for testing

I did try that, but it seemed not to work. I don’t understand why, maybe something to do with the triggering dependencies being included in the [weakdeps] section of Project.toml? Edit to add what didn’t work:

julia --project -e 'import Pkg; Pkg.add("NCDatasets"); Pkg.test()

failed because the extension was not loaded.

Edit to add:
Sorry, unless you mean something like

julia --project=temp -e 'import Pkg; Pkg.add("NCDatasets"); using NCDatasets; include("test/runtests.jl")

that might be a decent option acually…

5

If have push!(Base.LOAD_PATH, "@v#.#") inside my test/runtests.jl, to make this work on CI (together with the Pkg.add step in the global environment). It is not pretty, but hey, if it works, it works.

PS: I also have a try using NCDatasets ... in the test/runtests.jl.

6

I don’t really understand this point, what is wrong with users installing more dependencies if they want to test the package? To me it’s a fairly uncommon thing for users to run tests directly (as opposed to just relying on the package’s CI) and it seems fine to me in that case for them to install a bit more to run the tests manually.

7

I don’t really understand this point, what is wrong with users installing more dependencies if they want to test the package? To me it’s a fairly uncommon thing for users to run tests directly (as opposed to just relying on the package’s CI) and it seems fine to me in that case for them to install a bit more to run the tests manually.

In my case (scientific simulation software) essentially all users are also developers who will modify the source code. They then should run the tests to check that existing behaviour is not broken, but might not want to load all dependencies (for example because heavy dependencies make PackageCompiler.jl really slow, but we more-or-less have to use it to be able to submit MPI-parallelised jobs to a batch queue).

1 Like