Test.jl seems to have problems catching ArgParse.jl's exceptions

General Usage
, , , ,
1

Hi,

I’ve been implementing unit tests for my project, and generally I’m very happy with Test.jl, however I stumbled across a weird problem when trying to do tests for argument parsing. I handle it using ArgParse.jl and when running locally, @test_throws Exception parse_commandline() works fine. However, when I run it through Pkg.test(), I suddenly get an Error about a required option not being specified (I have 1 arg set as required for my project). I admit I’m not even sure if this is a problem with Test.jl or with ArgParse.jl (or my implementation :upside_down_face: ). Any suggestions would be highly welcome.

2

Hi! Are you testing by calling parse_args() with no arguments? This might interact with the system’s ARGS variable. The ArgParse package defines the function parse_args([args], settings). If you pass it a list of test arguments, I hope it behaves better. It shouldn’t at all interact with the test environment.

It might also be possible to copy the system ARGS to a saved variable and then write to the system ARGS, but that is explicitly forbidden by the runtime in other languages, like C, so I’m unsure about doing that in Julia.

You mentioned that the testing environment is different from running the program. Yes, Pkg.Test sets up a little environment and modifies the list of packages. If you wanted to understand that better, there is a package called TestEnv.jl that replicates this test environment setup in a place where you can read about it. It’s complicated.

Cheers,
Drew

1 Like
3

Hi Drew, thanks for your feedback! My function looks like this:

function parse_commandline()

    s = ArgParseSettings()
    @add_arg_table! s begin
        "-i"
        help = "Input file path."
        arg_type = String
        required = true
        "-o"
        help = "Output file name."
        arg_type = String
        default = "output"
        required = false
        "--fmt"
        help = "Output file format (`--fmt hdf5` or `--fmt csv`)."
        arg_type = String
        default = "csv"
        required = false
        "--plot-fmt"
        help = "Output file format for plots (e.g. `--plot-fmt png` or `--plot-fmt pdf`)."
        arg_type = String
        default = "png"
        required = false
...
        "--no-plots"
        help = "Turn off saving the visualisation of the results to files."
        action = :store_true
    end
    return parse_args(s)
end # parse_commandline

So you suggest to have

return parse_args(ARGS, s)

? :thinking: I’ve just tried it but I get identical problem with

@test_throws Exception parse_commandline()

I can get around it by adding

append!(ARGS, ["-i", "../data/example_image.dat"])

at the end of my runtests.jl, but it does not really seem like the most elegant solution.

4

For testing, would it work to do:

function parse_commandline(default_args=ARGS)
  s = ArgParseSettings()
  @add_arg_table! s begin
    ...
  end
  return parse_args(default_args, s)
end

Then pass your test arguments into parse_commandline(["--test-arg", "test_value"]). The issue you’re running into is that test environments can pack other args into the ARGS variable, so it’s good to stay away from it in tests.

5

Well, when I test the argparsing itself, I just start by doing

empty!(ARGS)

which deals with potential unwanted arguments.

6

That will cause problems if you start to use newer features in Test. The Test framework will use ARGS to support multithreading, and I think I’ve heard it uses ARGS already in some capacity.

Emptying ARGS will also make impossible a common use case: People add flags to their test suite to mark particular tests as --slow or --localonly so that they can run longer tests only when necessary.

I always like to make life easy by making things testable. There’s an older paper that talks about the SOCK principle. Testable code is simple, observable, controllable, and knowable. In this case, maximizing controllability is very doable.

Whatever you do, glad you’re thinking about testing. Enjoy.