Test-driven/test-heavy workflow?

General Usage
1

How do people set things up in order to run their test suite often (whether test-driven or not)?

Some options I’ve used:

  • Run test/runtests.jl from command line. Tends to take a long time, and requires global installation of test dependencies.
  • Run a function from test/runtests.jl in the REPL using Revise and includet. Works well enough, but also requires global installation of test dependencies (not a big deal, I guess).
  • Use the test command from Pkg. Works just swell, though it produces quite a lot of output in some cases, making it hard to (e.g.) compare output from different runs of the tests. (Of course, one might not normally have any output in the test suite, but I sometimes temporarily add @btime or the like during development…)

Other approaches? Thoughts/suggestions?

3 Likes
2

I do a variant of point 1: I make a folder called dev that’s gitignored, and make an environment there with the test dependencies and sometimes extra ones (like maybe plotting etc that isn’t a dependency of the main package but can be used along with it), and Pkg.develop the package itself. And then I activate this environment and include(“test/runtests.jl”) with revise.

3

How long it takes depends on the packages you need to load. But Pkg.test should take care of setting up the environment, regardless of what is in the global one.

That said, I do the following for packages with a lot of tests:

  1. have some setup code at the beginning of runtest.jl (load packages, utility functions)
  2. organize code into files that I then include from there, which I can evaluate more granularly
4

By command line, I meant the shell command line, not the Julia REPL, so Pkg.test isn’t involved in that scenario. The use of a long-running REPL is just really foreign to me, coming from other non-Matlab/-R languages, but I see I can’t really escape it :slight_smile: (And, yes, Pkg.test does deal with the setup if one does use the REPL – or maybe it can even be used in scripts – but has the minor output issue mentioned; if I could silence that, I’d be happy, I guess.)

5

This package helps with interpreting the output of tests (and other stuff)

2 Likes
6

Yeah, that’s a great package. (I considered it before putting together DotTestSets.jl.) It’s a bit orthogonal to what I’m thinking about, though, which is how to run the tests. It seems the default assumption here is that they’re run from the command line, which (with a semi-large code base, using JuMP and similar packages) can take too long to run as often as I’d like. (I’m sure it works well with the REPL as well, but the other disadvantages I listed still seem to be present?)

Anyway, thanks for the suggestion!

7

A possible solution, I guess, is to use

Pkg.DEFAULT_IO[] = devnull

Then you only get the test output, and not (e.g.) the list of transitive dependencies.

Of course, silencing Pkg is probably a very bad idea if you’re going to use it for anything else (and possibly not a good idea even for this use, for all I know). I suppose one could just have a separate REPL (e.g., in the Pkg REPL mode) where this is set.

I also tried the following, which didn’t work; I’m not sure if it’s supposed to? (Maybe not all calls respect ctx or ctx.io? There are at least some raw prints in there…)

import Pkg
import Pkg.Types: Context, PackageSpec
Pkg.test(Context(io=devnull), PackageSpec[])

I mean, it runs, but all output is still printed. :man_shrugging: