Test-specific dependencies

General Usage
1

In our package, we have a number of dependencies used for testing only. For end users of our package, we don’t wish to enforce those dependencies. It seems that the suggested way of achieving this is using Test-specific dependencies.

The problem with that technique is that, as far as I can tell, it only works when running the test suite through Pkg.test(), by creating a new temporary environment. But in our package, we have various ways of testing our code: the standard test suite, a script that allows testing a subset of the code, more extensive benchmarks, plotting the output, etc. These currently live as independent scripts in the test/ directory, e.g. test/runbenchmark.jl.

Now we’re wondering what the best way is to have a separate set of dependencies for those scripts only? Ideally an independent set of dependencies per script (in particular, we don’t want heavy dependencies like plotting unless really needed). One way is to programmatically add dependencies in the scripts themselves, so for example runbenchmark.jl could start with:


haskey(Pkg.installed(), "BenchmarkTools") || Pkg.add("BenchmarkTools")

A major disadvantage is that it modifies the Project.toml file. Any other ideas?

2

Seems like a perfect opportunity to use your own project files for this. See e.g. how you can use this for doc-dependencies PSA: Use a project for building your docs

2 Likes
3

Brilliant, thanks!! Using that technique, and adding a dependency on the parent project by path "..", is exactly what I was looking for.

4

So, if I make the test directory its own project, with its own Project.toml file, and still want Pkg.test() to work, I came up with the following runtests.jl:

push!(LOAD_PATH, "@stdlib")
using Pkg
Pkg.activate(@__DIR__)

# run tests here...

Is there a less hacky way?

5

One problem with this approach: I don’t find a way to use a module in the same directory as the Project.toml file. Let’s say I have this hierarchy:

.
├── Manifest.toml
├── Project.toml
├── src
│   └── MyPackage.jl
└── test
    ├── Manifest.toml
    ├── Project.toml
    ├── TestModule.jl
    └── runtests.jl

I can’t do using TestModule in runtests.jl. Even if I do push!(LOAD_PATH, @__DIR__), it still complains with:

ERROR: LoadError: ArgumentError: Package TestModule not found in current path:
- Run `Pkg.add("TestModule")` to install the TestModule package.

Any idea on how to accomplish this? If I move TestModule.jl to a subdirectory under test, and add that to the LOAD_PATH, it works, but I’d prefer to have it all in the same directory.

6

Do you want Pkg.test to run these extra test scripts? The easiest is probably to start a new julia process, something like

run(`$(Base.julia_cmd()) --project=$(@__DIR__) -e 'using Pkg; Pkg.instantiate(); include("extra_tests.jl")'`)

or if you just want this to run on e.g. CI you define your script to be something like

- julia -e 'using Pkg; Pkg.test()' # execute normal test suite
- julia --project=test/ -e 'using Pkg; Pkg.instantiate(); include("extra_tests.jl")'
7

Ultimately, this is probably something that will be solved by

https://github.com/JuliaLang/Pkg.jl/issues/624

Until then, we are stuck with the extra projects in sub-directories solution.

8

Thanks!

The idea would be to have Pkg.test() in the top-level project run the default test suite only, since people are used to running this command to run the unit tests, and would expect this to work.

For more advanced test scripts, I’d be ok with a different syntax. But importantly, I’d like at least some of these scripts to have the exact same dependencies as the default test suite, without having to repeat these dependencies in several project files. (We have for example one script that runs a subset of the default test suite.) I don’t think your second suggestion allows that, since it’d use the dependencies defined in the top-level Project.toml file to run the default tests?

For your first suggestion, I still get Package Pkg not found in current path. It also doesn’t like my startup.jl. The below works, but doesn’t feel any less hacky than the solution I posted above:

run(`$(Base.julia_cmd()) --startup-file=no --project=$(@__DIR__) -e
    'push!(LOAD_PATH, "@stdlib");
     using Pkg;
     Pkg.instantiate();
     include("extra_tests.jl")'`)