Are you looking for something more that Pkg.test("Package Name") for running tests? Running subsets of tests might be more difficult , because there is no default testing framework people are supposed to use. (And that seems to be a good idea, otherwise we wouldn’t necessarily had FactCheck -> BaseTestNext).
Do you have an example for other languages, that have something like this? It seems to me that this is mostly a problem that is solved by frameworks in other languages (Rails vs Ruby).
Also check out my new TestSetExtensions package (https://github.com/ssfrr/TestSetExtensions.jl). It has a way to selectively run different test files without assuming you’re writing your tests with anything except the basic Base.Test stuff from 0.5 (or BaseTestNext on 0.4).
I think the next steps here are to converge how tests are run for packages and Base Julia. @kslimes has done a lot of work towards this with a series of epic changes to make Julia’s base tests use the new test stuff (originally written by Iain Dunning). Base runs lots of tests in parallel; should packages automatically run their various test files in parallel? What are some of the other differences at this point?
I think base Julia’s test/math.jl and test/libgit2.jl give a good example of using test sets. test/runtests.jl “farms out” the test files to parallel workers and gathers some performance information along with the results (it’s also a cool example of using @async, I think).
Currently most packages don’t have enough tests that they need the kind of parallel testing functionality that Base Julia has, but some probably do already and more and more will – we might as well make it a built-in standard and use that in Base as well as packages. That’s what’s being suggested. (So what Katie said, in other words.)
Oh, I didn’t know you got multiple workers on Travis! I already have everything divided into testsets, so I assume I can just @spawn each testset expression (mine are all independent)? How do you get the number of processes to make on Travis (NUM_CORES)?
Here’s a snippet of what it looks like:
Since every testset is independent, they could all run at the same time.
Showing how to do parallel testing on Travis would be a really nice blog post that could really help development times!