Standard ways to run common commands

Internals & Design
1

Over the course of developing a project, there are certain commands that get executed all the time.

These include running:

  • Tests
  • Subsets of tests
  • The application’s main function

How can we introduce standard ways to perform these functions?

Also, a standard way to run functions would give:

  • people a nice list of hooks to plug their initialization code.
  • a way to warn people that their packages don’t meet REQUIRE specifications and other stuff
2

Are you looking for something more that Pkg.test("Package Name") for running tests? Running subsets of tests might be more difficult , because there is no default testing framework people are supposed to use. (And that seems to be a good idea, otherwise we wouldn’t necessarily had FactCheck → BaseTestNext).

Do you have an example for other languages, that have something like this? It seems to me that this is mostly a problem that is solved by frameworks in other languages (Rails vs Ruby).

3

+1 It’s a real pain to have to comment out the tests you want to run individually when developing a package.

Something like Pkg.test("MyModule","TestSet1") would be fantastic.

5 Likes
4

For many automation tasks, you can just write a function that does what you want. I wrote the following package to help with automating routine activities:

It’s like the make utility in that code runs based on updated dependencies. It’s focused more on analysis activities rather than file-based manipulations.

2 Likes
5

Also check out my new TestSetExtensions package (https://github.com/ssfrr/TestSetExtensions.jl). It has a way to selectively run different test files without assuming you’re writing your tests with anything except the basic Base.Test stuff from 0.5 (or BaseTestNext on 0.4).

</shameless plug>

-s

1 Like
6

I think the next steps here are to converge how tests are run for packages and Base Julia. @kslimes has done a lot of work towards this with a series of epic changes to make Julia’s base tests use the new test stuff (originally written by Iain Dunning). Base runs lots of tests in parallel; should packages automatically run their various test files in parallel? What are some of the other differences at this point?

2 Likes
8

Hi Stefan,

Can you elaborate on this a bit? What files give the flavor of what you’re talking about most? Maybe, show:

  • a few good examples from Juila’s base tests
  • some repos that you think reinvented the wheel too much (and highlight the need for standardization)
9

Hello @djsegal,

I think base Julia’s test/math.jl and test/libgit2.jl give a good example of using test sets. test/runtests.jl “farms out” the test files to parallel workers and gathers some performance information along with the results (it’s also a cool example of using @async, I think).

Does that answer your first question?

10

Currently most packages don’t have enough tests that they need the kind of parallel testing functionality that Base Julia has, but some probably do already and more and more will – we might as well make it a built-in standard and use that in Base as well as packages. That’s what’s being suggested. (So what Katie said, in other words.)

11

Oh, I didn’t know you got multiple workers on Travis! I already have everything divided into testsets, so I assume I can just @spawn each testset expression (mine are all independent)? How do you get the number of processes to make on Travis (NUM_CORES)?

Here’s a snippet of what it looks like:

Since every testset is independent, they could all run at the same time.

Showing how to do parallel testing on Travis would be a really nice blog post that could really help development times!

12

Are there plans to make Julia’s test infrastructure closer in feel to other languages?

Maybe @testset would benefit from some flags, including:

  • focus: true // only run this test
  • skip: true // skip this test
  • speed: slow // allow skipping these tests
  • type: solver // similar to speed but w/ file type

// note: these were just picked up off a quick scrub of rspec’s docs

edit: maybe allow timeout setting also (see this post)

2 Likes