# Test-driven/test-heavy workflow?

**URL:** <https://discourse.julialang.org/t/test-driven-test-heavy-workflow/35173>\
**Category:** General Usage\
**Tags:** testing\
**Created:** [February 26, 2020, 2:45pm UTC](https://discourse.julialang.org/t/test-driven-test-heavy-workflow/35173 "2020-02-26T14:45:05Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![mlhetland](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mlhetland/32/283_2.png) [@mlhetland](https://discourse.julialang.org/u/mlhetland)\
**Post date:** [February 26, 2020, 2:45pm UTC](https://discourse.julialang.org/t/test-driven-test-heavy-workflow/35173/1 "2020-02-26T14:45:05Z")

</div>

How do people set things up in order to run their test suite often (whether test-driven or not)?

Some options I’ve used:

- Run `test/runtests.jl` from command line. Tends to take a long time, and requires global installation of test dependencies.
- Run a function from `test/runtests.jl` in the REPL using Revise and `includet`. Works well enough, but also requires global installation of test dependencies (not a big deal, I guess).
- Use the `test` command from `Pkg`. Works just swell, though it produces quite a lot of output in some cases, making it hard to (e.g.) compare output from different runs of the tests. (Of course, one might not normally have any output in the test suite, but I sometimes temporarily add `@btime` or the like during development…)

Other approaches? Thoughts/suggestions?

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [February 26, 2020, 3:17pm UTC](https://discourse.julialang.org/t/test-driven-test-heavy-workflow/35173/2 "2020-02-26T15:17:43Z")

</div>

I do a variant of point 1: I make a folder called `dev` that’s gitignored, and make an environment there with the test dependencies and sometimes extra ones (like maybe plotting etc that isn’t a dependency of the main package but can be used along with it), and `Pkg.develop` the package itself. And then I activate this environment and `include(“test/runtests.jl”)` with revise.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [February 26, 2020, 5:51pm UTC](https://discourse.julialang.org/t/test-driven-test-heavy-workflow/35173/3 "2020-02-26T17:51:35Z")

</div>

> [@mlhetland](#):
>
> Tends to take a long time, and requires global installation of test dependencies.

How long it takes depends on the packages you need to load. But `Pkg.test` should take care of setting up the environment, regardless of what is in the global one.

That said, I do the following for packages with a lot of tests:

1. have some setup code at the beginning of `runtest.jl` (load packages, utility functions)
2. organize code into files that I then `include` from there, which I can evaluate more granularly

---

<div class="post-metadata">

**Author:** ![mlhetland](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mlhetland/32/283_2.png) [@mlhetland](https://discourse.julialang.org/u/mlhetland)\
**Post date:** [February 26, 2020, 6:19pm UTC](https://discourse.julialang.org/t/test-driven-test-heavy-workflow/35173/4 "2020-02-26T18:19:37Z")

</div>

> [@Tamas\_Papp](#):
>
> How long it takes depends on the packages you need to load. But `Pkg.test` should take care of setting up the environment, regardless of what is in the global one.

By command line, I meant the shell command line, not the Julia REPL, so `Pkg.test` isn’t involved in that scenario. The use of a long-running REPL is just really foreign to me, coming from other non-Matlab/-R languages, but I see I can’t really escape it 🙂 (And, yes, `Pkg.test` does deal with the setup if one does use the REPL – or maybe it can even be used in scripts – but has the minor output issue mentioned; if I could silence that, I’d be happy, I guess.)

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [February 27, 2020, 8:27am UTC](https://discourse.julialang.org/t/test-driven-test-heavy-workflow/35173/5 "2020-02-27T08:27:02Z")

</div>

This package helps with interpreting the output of tests (and other stuff)

> **[GitHub - ssfrr/TestSetExtensions.jl: Extensions to Julia's Base.Test](https://github.com/ssfrr/TestSetExtensions.jl)**
>
> Extensions to Julia's Base.Test. Contribute to ssfrr/TestSetExtensions.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![mlhetland](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mlhetland/32/283_2.png) [@mlhetland](https://discourse.julialang.org/u/mlhetland)\
**Post date:** [February 27, 2020, 8:33am UTC](https://discourse.julialang.org/t/test-driven-test-heavy-workflow/35173/6 "2020-02-27T08:33:37Z")

</div>

Yeah, that’s a great package. (I considered it before putting together [`DotTestSets.jl`](https://github.com/mlhetland/DotTestSets.jl).) It’s a bit orthogonal to what I’m thinking about, though, which is how to run the tests. It seems the default assumption here is that they’re run from the command line, which (with a semi-large code base, using JuMP and similar packages) can take too long to run as often as I’d like. (I’m sure it works well with the REPL as well, but the other disadvantages I listed still seem to be present?)

Anyway, thanks for the suggestion!

---

<div class="post-metadata">

**Author:** ![mlhetland](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mlhetland/32/283_2.png) [@mlhetland](https://discourse.julialang.org/u/mlhetland)\
**Post date:** [February 27, 2020, 11:40am UTC](https://discourse.julialang.org/t/test-driven-test-heavy-workflow/35173/7 "2020-02-27T11:40:57Z")

</div>

A possible solution, I guess, is to use

```julia
Pkg.DEFAULT_IO[] = devnull

```

Then you only get the test output, and not (e.g.) the list of transitive dependencies.

Of course, silencing `Pkg` is probably a very bad idea if you’re going to use it for anything else (and possibly not a good idea even for this use, for all I know). I suppose one could just have a separate REPL (e.g., in the `Pkg` REPL mode) where this is set.

I also tried the following, which didn’t work; I’m not sure if it’s supposed to? (Maybe not all calls respect `ctx` or `ctx.io`? There are at least some raw `print`s in there…)

```julia
import Pkg
import Pkg.Types: Context, PackageSpec
Pkg.test(Context(io=devnull), PackageSpec[])

```

I mean, it runs, but all output is still printed. 🤷‍♂️