I would propose the following rules for organizing files in test directories:
any test/*.jl file is a test file
for any directory test/foo/
if test/foo.jl exists, test/foo/ contains support files for test/foo.jl and is ignored
otherwise test/foo is considered to be a test subdirectory and should be organized similarly to a top-level test directory
The main exception would have to be the top-level test/runtest.jl file, but I’d actually like to eliminate that if possible since we want to have a common standard way of organizing and running tests. We might want some testing metadata in a .toml file either in test or in project configuration files.
Also, is this for packages or base julia?
If it’s going to be used for packages, how should this deal with custom logic needed by packages (like running setup/cleanup code).
Are the files going to be wrapped in it’s own module (which will create quite a bit of overhead)?
I think the runtests.jl + include all test files + throw exception on test error is a bit inflexible, epsecially when it comes to pretty printing of test results. Instead having files (and tests) getting “auto discovered” (á la Good Integration Practices — pytest documentation) seems to easier allow for a “test runner” to run tests, gather results, and pretty print them.
Oh, I think I read the first point as any test/*.jl file is a test as in each file has only one test, which sounds a bit crazy.
The proposed organization seems good but I think we still want to keep runtests.jl since it’s more flexible. As I already mentioned in https://github.com/JuliaLang/julia/pull/19567, I think we should have a helper function to discover and run tests following a certain directory structure. Any further customization will be the most flexible if it is done using a wrapper script (i.e. runtests.jl) around such function.
The problem with that is that you want to be able to do things like list and describe what sets of tests you can select at the top level. If runtests defines that imperatively instead of declaratively, then that also needs to be defined, which is annoying.
# Do some initialization
function choosetests(tests)
# Fancy custom logic of picking tests
end
Test.runtests(choosetests) # possibly with other flags
# Do some clean up
```
And `Test.runtests` can just discover the test names and feed it to the callback for further customization.
Using any test files layout we decide here? I think the one you proposed above should work. The Test.runtest() function can either use source_file() to figure out the test directory or let the user pass it in directly.
At least when the user want to customize something. We can certainly assume the standard test file layout without customization if no runtests.jl file is found so that the majority of packages that doesn’t need any customization doesn’t need the runtests.jl file.
I don’t understand the reason for needing more flexibility. Making this standard and not require any test/runtests.jl file at all seems better to me. We can also have a default behavior which can be overridden by the presence of test/runtests.jl instead of requiring every project to have the file when most of them are identical.
That’s exactly what I said. I mainly don’t think we should use a data file for customization for tests.
Also, in additional to allowing more customization, exposing the feature as a function (to be called by packages that provides runtests.jl) should also make it easier to run the tests from other environment. It’s very useful for debugging to be able to run the tests without starting a new process.
And I think we should try to use the same code for base too so we should make it possible to support the logic used by base tests. (Even if we don’t use it for base, the base one should be a good reference to see what logics are possibly necessary)
Things that the packages might want to customize for choosing tests can include.
Arch/platform check (in principle this can be specified in other ways)
Tests that depend on certain build flags, e.g. USE_GPL_LIBS in base or optional dependencies for some packages.
Tests that depend on test environments (travis environment variables, or net_on or jl_running_on_valgrind)
I think it’ll be the easiest to specify these logic in julia code and it’ll be great if the package (and base) that needs this level of customization can still use the same tests discovery and cmdline argument handling.
I think we should try to use the same code for base too so we should make it possible to support the logic used by base tests.
100% agree. What I’m pushing for is a single common infrastructure for testing base, packages and other projects. Base is a good initial test case since it’s got a lot of tests and pushes the testing infrastructure quite hard.
It would also be good to allow support files that are shared by multiple test files (unless I’m missing it). I often have a support.jl or util.jlor similar that’s included at the top of my runtests.jl and provides types or functions that are used by multiple test files.
I’m not sure I agree with this. It seems like Base pushes the testing infrastructure way harder than almost any packages, and in that sense is pretty dissimilar from the average package use-case.
In general I’m not sold that we need much more than what we have now. I think the test/runtests.jl rule provides a simple entry point for testing, @testset/@test provide nice building blocks for writing tests, and packages can provide other test functionality like pretty-printing output, reporting to CI tools and dashboards, test file discovery, etc. Maybe I’ve just settled into a comfortable local maximum, but I’m pretty happy with my testing setup.
AFAICT the main benefit of the declarative approach and/or enforcing a file structure is the ability to list available tests, but for most packages that could be as simple as ls MyPkg/test.
Big caveat is that you see a much greater variety of Julia users than I do, so I hold my opinion lightly.