# RFC: test file organization proposal

**URL:** <https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170>\
**Category:** Internals & Design\
**Tags:** proposal, testing\
**Created:** [December 27, 2016, 11:11pm UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170 "2016-12-27T23:11:46Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 27, 2016, 11:11pm UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/1 "2016-12-27T23:11:46Z")

</div>

I would propose the following rules for organizing files in `test` directories:

- any `test/*.jl` file is a test file
- for any directory `test/foo/`
  - if `test/foo.jl` exists, `test/foo/` contains support files for `test/foo.jl` and is ignored
  - otherwise `test/foo` is considered to be a test subdirectory and should be organized similarly to a top-level `test` directory

The main exception would have to be the top-level `test/runtest.jl` file, but I’d actually like to eliminate that if possible since we want to have a common standard way of organizing and running tests. We might want some testing metadata in a `.toml` file either in `test` or in project configuration files.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [December 27, 2016, 11:22pm UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/2 "2016-12-27T23:22:09Z")

</div>

Why having so many small files?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [December 27, 2016, 11:25pm UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/3 "2016-12-27T23:25:59Z")

</div>

Also, is this for packages or base julia?  
If it’s going to be used for packages, how should this deal with custom logic needed by packages (like running setup/cleanup code).  
Are the files going to be wrapped in it’s own module (which will create quite a bit of overhead)?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [December 27, 2016, 11:40pm UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/4 "2016-12-27T23:40:34Z")

</div>

I think the `runtests.jl` + `include` all test files + throw exception on test error is a bit inflexible, epsecially when it comes to pretty printing of test results. Instead having files (and tests) getting “auto discovered” (á la [Good Integration Practices — pytest documentation](http://doc.pytest.org/en/latest/goodpractices.html#test-discovery)) seems to easier allow for a “test runner” to run tests, gather results, and pretty print them.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [December 27, 2016, 11:45pm UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/5 "2016-12-27T23:45:29Z")

</div>

Oh, I think I read the first point as `any test/*.jl file is a test` as in each file has only one test, which sounds a bit crazy.  
The proposed organization seems good but I think we still want to keep `runtests.jl` since it’s more flexible. As I already mentioned in [https://github.com/JuliaLang/julia/pull/19567](https://github.com/JuliaLang/julia/pull/19567), I think we should have a helper function to discover and run tests following a certain directory structure. Any further customization will be the most flexible if it is done using a wrapper script (i.e. `runtests.jl`) around such function.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 28, 2016, 12:03am UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/6 "2016-12-28T00:03:18Z")

</div>

The problem with that is that you want to be able to do things like list and describe what sets of tests you can select at the top level. If runtests defines that imperatively instead of declaratively, then that also needs to be defined, which is annoying.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [December 28, 2016, 12:12am UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/7 "2016-12-28T00:12:36Z")

</div>

Won’t a [choosetest callback](https://github.com/JuliaLang/julia/pull/19567#discussion_r92528978) do it without having to list the tests manually?

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 28, 2016, 12:14am UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/8 "2016-12-28T00:14:48Z")

</div>

I don’t think that addresses knowing what sets of tests are available, does it?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [December 28, 2016, 12:17am UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/9 "2016-12-28T00:17:54Z")

</div>

The `runtests.jl` will look like

````julia

# Do some initialization

function choosetests(tests)
    # Fancy custom logic of picking tests
end

Test.runtests(choosetests) # possibly with other flags

# Do some clean up

```

And `Test.runtests` can just discover the test names and feed it to the callback for further customization.
````

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 28, 2016, 12:18am UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/10 "2016-12-28T00:18:51Z")

</div>

How does one find out what sets of tests are available?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [December 28, 2016, 12:20am UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/11 "2016-12-28T00:20:43Z")

</div>

Using any test files layout we decide here? I think the one you proposed above should work. The `Test.runtest()` function can either use `source_file()` to figure out the test directory or let the user pass it in directly.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 28, 2016, 12:21am UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/12 "2016-12-28T00:21:48Z")

</div>

So are you agreeing with me or disagreeing?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [December 28, 2016, 12:22am UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/13 "2016-12-28T00:22:25Z")

</div>

I agree with the organization but I think we should keep `runtests.jl` as the test entry point.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [December 28, 2016, 12:24am UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/14 "2016-12-28T00:24:35Z")

</div>

At least when the user want to customize something. We can certainly assume the standard test file layout without customization if no `runtests.jl` file is found so that the majority of packages that doesn’t need any customization doesn’t need the `runtests.jl` file.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 28, 2016, 12:25am UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/15 "2016-12-28T00:25:36Z")

</div>

I don’t understand the reason for needing more flexibility. Making this standard and not require any `test/runtests.jl` file at all seems better to me. We can also have a default behavior which can be overridden by the presence of `test/runtests.jl` instead of requiring every project to have the file when most of them are identical.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [December 28, 2016, 12:31am UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/16 "2016-12-28T00:31:50Z")

</div>

> [@StefanKarpinski](#):
>
> We can also have a default behavior which can be overridden by the presence of test/runtests.jl instead of requiring every project to have the file when most of them are identical.

That’s exactly what I said. I mainly don’t think we should use a data file for customization for tests.

Also, in additional to allowing more customization, exposing the feature as a function (to be called by packages that provides `runtests.jl`) should also make it easier to run the tests from other environment. It’s very useful for debugging to be able to run the tests without starting a new process.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [December 28, 2016, 12:49am UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/17 "2016-12-28T00:49:03Z")

</div>

And I think we should try to use the same code for base too so we should make it possible to support the logic used by base tests. (Even if we don’t use it for base, the base one should be a good reference to see what logics are possibly necessary)

Things that the packages might want to customize for choosing tests can include.

- Arch/platform check (in principle this can be specified in other ways)
- Tests that depend on certain build flags, e.g. `USE_GPL_LIBS` in base or optional dependencies for some packages.
- Tests that depend on test environments (travis environment variables, or `net_on` or `jl_running_on_valgrind`)

I think it’ll be the easiest to specify these logic in julia code and it’ll be great if the package (and base) that needs this level of customization can still use the same tests discovery and cmdline argument handling.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 28, 2016, 2:47pm UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/18 "2016-12-28T14:47:47Z")

</div>

> I think we should try to use the same code for base too so we should make it possible to support the logic used by base tests.

100% agree. What I’m pushing for is a single common infrastructure for testing base, packages and other projects. Base is a good initial test case since it’s got a lot of tests and pushes the testing infrastructure quite hard.

---

<div class="post-metadata">

**Author:** ![ssfrr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ssfrr/32/3736_2.png) [@ssfrr](https://discourse.julialang.org/u/ssfrr)\
**Post date:** [December 28, 2016, 10:44pm UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/19 "2016-12-28T22:44:06Z")

</div>

> [@StefanKarpinski](#):
>
> - any test/\*.jl file is a test file
> - for any directory test/foo/
> - if test/foo.jl exists, test/foo/ contains support files for test/foo.jl and is ignored
> - otherwise test/foo is considered to be a test subdirectory and should be organized similarly to a top-level test directory

It would also be good to allow support files that are shared by multiple test files (unless I’m missing it). I often have a `support.jl` or `util.jl`or similar that’s included at the top of my `runtests.jl` and provides types or functions that are used by multiple test files.

> [@StefanKarpinski](#):
>
> Base is a good initial test case since it’s got a lot of tests and pushes the testing infrastructure quite hard.

I’m not sure I agree with this. It seems like Base pushes the testing infrastructure way harder than almost any packages, and in that sense is pretty dissimilar from the average package use-case.

In general I’m not sold that we need much more than what we have now. I think the `test/runtests.jl` rule provides a simple entry point for testing, `@testset`/`@test` provide nice building blocks for writing tests, and packages can provide other test functionality like pretty-printing output, reporting to CI tools and dashboards, test file discovery, etc. Maybe I’ve just settled into a comfortable local maximum, but I’m pretty happy with [my testing setup](https://github.com/JuliaAudio/SampledSignals.jl/blob/38289706e65478f3b0af10c9a73cbba6544f105e/test/runtests.jl).

AFAICT the main benefit of the declarative approach and/or enforcing a file structure is the ability to list available tests, but for most packages that could be as simple as `ls MyPkg/test`.

Big caveat is that you see a much greater variety of Julia users than I do, so I hold my opinion lightly.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 28, 2016, 11:07pm UTC](https://discourse.julialang.org/t/rfc-test-file-organization-proposal/1170/20 "2016-12-28T23:07:12Z")

</div>

> [@ssfrr](#):
>
> It would also be good to allow support files that are shared by multiple test files (unless I’m missing it). I often have a support.jl or util.jlor similar that’s included at the top of my runtests.jl and provides types or functions that are used by multiple test files.

In Stefan’s proposal, you’d put these under `test/runtests` (assuming you have a `test/runtests.jl` file)