How do people set things up in order to run their test suite often (whether test-driven or not)?
Some options I’ve used:
test/runtests.jl from command line. Tends to take a long time, and requires global installation of test dependencies.
- Run a function from
test/runtests.jl in the REPL using Revise and
includet. Works well enough, but also requires global installation of test dependencies (not a big deal, I guess).
- Use the
test command from
Pkg. Works just swell, though it produces quite a lot of output in some cases, making it hard to (e.g.) compare output from different runs of the tests. (Of course, one might not normally have any output in the test suite, but I sometimes temporarily add
@btime or the like during development…)
Other approaches? Thoughts/suggestions?
I do a variant of point 1: I make a folder called
dev that’s gitignored, and make an environment there with the test dependencies and sometimes extra ones (like maybe plotting etc that isn’t a dependency of the main package but can be used along with it), and
Pkg.develop the package itself. And then I activate this environment and
include(“test/runtests.jl”) with revise.
How long it takes depends on the packages you need to load. But
Pkg.test should take care of setting up the environment, regardless of what is in the global one.
That said, I do the following for packages with a lot of tests:
- have some setup code at the beginning of
runtest.jl (load packages, utility functions)
- organize code into files that I then
include from there, which I can evaluate more granularly
By command line, I meant the shell command line, not the Julia REPL, so
Pkg.test isn’t involved in that scenario. The use of a long-running REPL is just really foreign to me, coming from other non-Matlab/-R languages, but I see I can’t really escape it (And, yes,
Pkg.test does deal with the setup if one does use the REPL – or maybe it can even be used in scripts – but has the minor output issue mentioned; if I could silence that, I’d be happy, I guess.)
This package helps with interpreting the output of tests (and other stuff)
Yeah, that’s a great package. (I considered it before putting together
DotTestSets.jl.) It’s a bit orthogonal to what I’m thinking about, though, which is how to run the tests. It seems the default assumption here is that they’re run from the command line, which (with a semi-large code base, using JuMP and similar packages) can take too long to run as often as I’d like. (I’m sure it works well with the REPL as well, but the other disadvantages I listed still seem to be present?)
Anyway, thanks for the suggestion!
A possible solution, I guess, is to use
Pkg.DEFAULT_IO = devnull
Then you only get the test output, and not (e.g.) the list of transitive dependencies.
Of course, silencing
Pkg is probably a very bad idea if you’re going to use it for anything else (and possibly not a good idea even for this use, for all I know). I suppose one could just have a separate REPL (e.g., in the
Pkg REPL mode) where this is set.
I also tried the following, which didn’t work; I’m not sure if it’s supposed to? (Maybe not all calls respect
ctx.io? There are at least some raw
prints in there…)
import Pkg.Types: Context, PackageSpec
I mean, it runs, but all output is still printed.