Standard ways to run common commands


#1

Over the course of developing a project, there are certain commands that get executed all the time.

These include running:

  • Tests
  • Subsets of tests
  • The application’s main function

How can we introduce standard ways to perform these functions?


Also, a standard way to run functions would give:

  • people a nice list of hooks to plug their initialization code.
  • a way to warn people that their packages don’t meet REQUIRE specifications and other stuff

#2

Are you looking for something more that Pkg.test("Package Name") for running tests? Running subsets of tests might be more difficult , because there is no default testing framework people are supposed to use. (And that seems to be a good idea, otherwise we wouldn’t necessarily had FactCheck -> BaseTestNext).

Do you have an example for other languages, that have something like this? It seems to me that this is mostly a problem that is solved by frameworks in other languages (Rails vs Ruby).


#3

+1 It’s a real pain to have to comment out the tests you want to run individually when developing a package.

Something like Pkg.test("MyModule","TestSet1") would be fantastic.


#4

For many automation tasks, you can just write a function that does what you want. I wrote the following package to help with automating routine activities:

It’s like the make utility in that code runs based on updated dependencies. It’s focused more on analysis activities rather than file-based manipulations.


#5

Also check out my new TestSetExtensions package (https://github.com/ssfrr/TestSetExtensions.jl). It has a way to selectively run different test files without assuming you’re writing your tests with anything except the basic Base.Test stuff from 0.5 (or BaseTestNext on 0.4).

</shameless plug>

-s


#6

I think the next steps here are to converge how tests are run for packages and Base Julia. @kslimes has done a lot of work towards this with a series of epic changes to make Julia’s base tests use the new test stuff (originally written by Iain Dunning). Base runs lots of tests in parallel; should packages automatically run their various test files in parallel? What are some of the other differences at this point?


#8

Hi Stefan,

Can you elaborate on this a bit? What files give the flavor of what you’re talking about most? Maybe, show:

  • a few good examples from Juila’s base tests
  • some repos that you think reinvented the wheel too much (and highlight the need for standardization)

#9

Hello @djsegal,

I think base Julia’s test/math.jl and test/libgit2.jl give a good example of using test sets. test/runtests.jl “farms out” the test files to parallel workers and gathers some performance information along with the results (it’s also a cool example of using @async, I think).

Does that answer your first question?


#10

Currently most packages don’t have enough tests that they need the kind of parallel testing functionality that Base Julia has, but some probably do already and more and more will – we might as well make it a built-in standard and use that in Base as well as packages. That’s what’s being suggested. (So what Katie said, in other words.)


#11

Oh, I didn’t know you got multiple workers on Travis! I already have everything divided into testsets, so I assume I can just @spawn each testset expression (mine are all independent)? How do you get the number of processes to make on Travis (NUM_CORES)?

Here’s a snippet of what it looks like:

Since every testset is independent, they could all run at the same time.

Showing how to do parallel testing on Travis would be a really nice blog post that could really help development times!


#12

Are there plans to make Julia’s test infrastructure closer in feel to other languages?

Maybe @testset would benefit from some flags, including:

  • focus: true // only run this test
  • skip: true // skip this test
  • speed: slow // allow skipping these tests
  • type: solver // similar to speed but w/ file type

// note: these were just picked up off a quick scrub of rspec’s docs


edit: maybe allow timeout setting also (see this post)


Workflow for Julia scripts
Delivering a Julia application within a package