Trying to figure out what testing etc libraries I should be using for julia

General Usage
1
  1. I’m trying to figure out what testing etc libraries I should be using for julia, especially for property-based testing. I’m aware of SafeTestSets.jl, TestSetExtensions.jl, ReferenceTests.jl, UnitTestDesign.jl, RandomizedPropertyTest.jl Are there any other libraries I should look into?
  2. Is the main way to program with contracts in Julia to use ‘traits’?

Note that the original poster on Slack cannot see your response here on Discourse. Consider transcribing the appropriate answer back to Slack, or pinging the poster here on Discourse so they can follow this thread.
(Original message :slack:) (More Info)

2 Likes
2

I’ll now transcribe the responses, since they were interesting (though I feel like this transcription should be automatable):

Valentin:

  • I’m not aware of any package explicitly for property based testing (though I’ve worked on a prototype I’m not happy with)
  • In julia, traits are mostly used to encode extra information that’s not part of the type, to change or control dispatch more finely grained
    most of the time, regular dispatch and duck typing is enough though

Boston Gelwan: Development with Interface Packages · Invenia Blog

When I brought up RandomizedPropertyTest, Valentin pointed out that it has a serious shortcoming:

RandomizedPropertyTest.jl does type based generation and no shrinking, sadly :confused:
in julia we don’t really have the same kind of type information as Haskell, so type based shrinking doesn’t come for free for composite types
Integrated vs type based shrinking - Hypothesis

Drew Dolgert also volunteered a very nice summary of the testing libraries out there, and which I’ll paste in a reply below; Drew also mentioned a test reduction package that’s worth noting: see GitHub - maleadt/creduce_julia and ANN: Automatic test-case reduction and Bugs in Julia with AFL and C-Reduce · maleadt

Finally, one unit-testing framework that isn’t already in Drew’s notes, and which seems worth being aware of, is

4 Likes
3

Drew’s notes:

Extensions to Test Sets

  • XXX what’s up with this? Still in use?
  • nested, typed test sets
  • makes results pretty
  • group tests under a description
  • makes a unit test
  • skippable tests
  • Makes testsets less likely to interfere with each other
  • by wrapping them in a module
  • shows dots as tests complete, by hooking Test
  • shows colored dots
  • failing tests show diffs between found and expected
  • selectively run test files with @includetests
  • filter tests based on regular expressions against testset names
  • include tests found within source files
  • Makes JUnit XML test reports
  • adds @timedtestset
  • adds @constinferred to see if return value inferred from constant propagation
  • Watch code and test when it is updated, on a function-by-function basis

Separate testing frameworks

  • run specific tests
  • run test when files change
  • skip test modules
  • run a block only once in a loop
  • conditionally evaluate module
  • Write a Markdown description of behavior and pull out tests
  • Identifies tests, like functions, to collect them and then run.
  • like py.test
  • Puts tests in functions in modules
  • Literate programming tools (like Sweave) to write tests

Testing tools

Performance testing

  • enables nonfunctional performance testing inside CI tools
  • adds an include macro that runs tests in different process

Regression testing

  • compares values with values in files
  • Generates smaller testing examples.
  • Reduce size of suite while keeping behavior

Floating-point assessment

  • Run a function on an interval in order to find numerical accuracy problems.

BigFloat can set rounding mode, which is a common heuristic to
understand sensitivity of a calculation to numerical problems.

Mocking

  • mock a function
  • Mocking as a do-context
  • Mocking by annotating a function that you want to patch later

Mutation testing

  • Mutation testing
  • Fuzzer using concolic (concrete + symbolic) execution
  • Runs code, watches it run, uses that to make the next choice.

Generation of parameter values

  • Generates parameter values from the input domain
  • Weights random generation to emphasize edge cases
  • Like Python’s Hypothesis

https://gitlab.com/quf/randomizedpropertytest.jl

  • Randomized property testing
  • QuickCheck inspired
  • Factorial test designs
  • All-pairs, all-triples.

Code Coverage

  • Instrument code to get coverage
  • Instruments for smaller code coverage by line

Test specific tools

  • Regression testing of user interfaces to command-line sessions
  • A suite of matrices used for testing algorithms
  • Polynomials to solve for benchmarking
  • run javascript to test within Electron
  • Specific to testing differentiation tools
  • Images for testing
  • Record and replay http tests
  • Compare images for testing
  • Tests of optimization problems
  • Test automatic differentiation primitives
  • Test of random number generation
  • Test of POMDP models
  • Functions to test nonlinear optimization
  • Tests for nonlinear / polynomial solvers.
  • Data for tests of phylogenetic networks
  • Tests of the physical layer (OSI model)
  • Functions to optimize to test optimization
  • Biological file format specimens

Code rewriting

  • Insert behavior into the JIT
  • Work with Julia code and expressions
  • Symbolic parser generator
  • Pattern-matching metaprogramming
  • Tools to rewrite the AST

https://github.com/FluxML/IRTools.jl

  • Provides an intermediate representation to work with.
  • Works with Cassette.jl
8 Likes