# Trying to figure out what testing etc libraries I should be using for julia

**URL:** <https://discourse.julialang.org/t/trying-to-figure-out-what-testing-etc-libraries-i-should-be-using-for-julia/56429>\
**Category:** General Usage\
**Created:** [March 3, 2021, 6:12pm UTC](https://discourse.julialang.org/t/trying-to-figure-out-what-testing-etc-libraries-i-should-be-using-for-julia/56429 "2021-03-03T18:12:13Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![BridgeBot](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bridgebot/32/21491_2.png) [@BridgeBot](https://discourse.julialang.org/u/BridgeBot)\
**Post date:** [March 3, 2021, 6:12pm UTC](https://discourse.julialang.org/t/trying-to-figure-out-what-testing-etc-libraries-i-should-be-using-for-julia/56429/1 "2021-03-03T18:12:13Z")

</div>

1. I’m trying to figure out what testing etc libraries I should be using for julia, especially for property-based testing. I’m aware of SafeTestSets.jl, TestSetExtensions.jl, ReferenceTests.jl, UnitTestDesign.jl, RandomizedPropertyTest.jl Are there any other libraries I should look into?
2. Is the main way to program with contracts in Julia to use ‘traits’?

Note that the original poster on Slack cannot see your response here on Discourse. Consider _transcribing the appropriate answer back to Slack_, or pinging the poster here on Discourse so they can _follow this thread_.  
[(Original message :slack:)](https://julialang.slack.com/archives/C6A044SQH/p1614795113282400?thread_ts=1614795113.282400&cid=C6A044SQH) [(More Info)](https://github.com/JuliaCommunity/SlackBridge)

---

<div class="post-metadata">

**Author:** ![ymh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ymh/32/14650_2.png) [@ymh](https://discourse.julialang.org/u/ymh)\
**Post date:** [March 3, 2021, 9:55pm UTC](https://discourse.julialang.org/t/trying-to-figure-out-what-testing-etc-libraries-i-should-be-using-for-julia/56429/2 "2021-03-03T21:55:02Z")

</div>

I’ll now transcribe the responses, since they were interesting (though I feel like this transcription should be automatable):

Valentin:

- I’m not aware of any package explicitly for property based testing (though I’ve worked on a prototype I’m not happy with)
- In julia, traits are mostly used to encode extra information that’s not part of the type, to change or control dispatch more finely grained  
most of the time, regular dispatch and duck typing is enough though

Boston Gelwan: [Development with Interface Packages · Invenia Blog](https://invenia.github.io/blog/2020/11/06/interfacetesting/)

When I brought up RandomizedPropertyTest, Valentin pointed out that it has a serious shortcoming:

> RandomizedPropertyTest.jl does type based generation and no shrinking, sadly 😕  
> in julia we don’t really have the same kind of type information as Haskell, so type based shrinking doesn’t come for free for composite types  
> [Integrated vs type based shrinking - Hypothesis](https://hypothesis.works/articles/integrated-shrinking/)

Drew Dolgert also volunteered a very nice summary of the testing libraries out there, and which I’ll paste in a reply below; Drew also mentioned a test reduction package that’s worth noting: see [GitHub - maleadt/creduce\_julia](https://github.com/maleadt/creduce_julia) and [ANN: Automatic test-case reduction](https://discourse.julialang.org/t/ann-automatic-test-case-reduction/17700) and [Bugs in Julia with AFL and C-Reduce · maleadt](https://blog.maleadt.net/2018/11/16/julia_bugs/)

Finally, one unit-testing framework that isn’t already in Drew’s notes, and which seems worth being aware of, is

> **[GitHub - maleadt/XUnit.jl: XUnit.jl is a unit-testing framework for Julia.](https://github.com/maleadt/XUnit.jl)**
>
> XUnit.jl is a unit-testing framework for Julia. Contribute to maleadt/XUnit.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![ymh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ymh/32/14650_2.png) [@ymh](https://discourse.julialang.org/u/ymh)\
**Post date:** [March 6, 2021, 9:09pm UTC](https://discourse.julialang.org/t/trying-to-figure-out-what-testing-etc-libraries-i-should-be-using-for-julia/56429/3 "2021-03-06T21:09:18Z")

</div>

Drew’s notes:

# Extensions to Test Sets

> **[GitHub - apieum/Maracas.jl: julia testing helpers](https://github.com/apieum/Maracas.jl)**
>
> julia testing helpers

- XXX what’s up with this? Still in use?
- nested, typed test sets
- makes results pretty
- group tests under a description
- makes a unit test
- skippable tests

> **[GitHub - YingboMa/SafeTestsets.jl](https://github.com/YingboMa/SafeTestsets.jl)**
>
> Contribute to YingboMa/SafeTestsets.jl development by creating an account on GitHub.

- Makes testsets less likely to interfere with each other
- by wrapping them in a module

> **[GitHub - mlhetland/DotTestSets.jl: Custom Julia test sets with output similar to that...](https://github.com/mlhetland/DotTestSets.jl)**
>
> Custom Julia test sets with output similar to that of Python's unittest

- shows dots as tests complete, by hooking Test

> **[GitHub - ssfrr/TestSetExtensions.jl: Extensions to Julia's Base.Test](https://github.com/ssfrr/TestSetExtensions.jl)**
>
> Extensions to Julia's Base.Test

- shows colored dots
- failing tests show diffs between found and expected
- selectively run test files with @includetests

> **[GitHub - JuliaTesting/ReTest.jl: Testing framework for Julia](https://github.com/JuliaTesting/ReTest.jl)**
>
> Testing framework for Julia

- filter tests based on regular expressions against testset names
- include tests found within source files

> **[GitHub - JuliaTesting/TestReports.jl: Making JUnit style XML reports for your julia...](https://github.com/JuliaTesting/TestReports.jl)**
>
> Making JUnit style XML reports for your julia tests

- Makes JUnit XML test reports

> **[GitHub - Jutho/TestExtras.jl: Some useful additions to the Julia Test Standard...](https://github.com/Jutho/TestExtras.jl)**
>
> Some useful additions to the Julia Test Standard Library

- adds @timedtestset
- adds @constinferred to see if return value inferred from constant propagation

> **[GitHub - pengwyn/EyeOfRa.jl: Continuous testing while developing functions](https://github.com/pengwyn/EyeOfRa.jl)**
>
> Continuous testing while developing functions

- Watch code and test when it is updated, on a function-by-function basis

# Separate testing frameworks

> **[GitHub - wookay/Jive.jl: some useful steps in tests 👣](https://github.com/wookay/Jive.jl)**
>
> some useful steps in tests 👣

- run specific tests
- run test when files change
- skip test modules
- run a block only once in a loop
- conditionally evaluate module

> **[GitHub - MechanicalRabbit/NarrativeTest.jl: Julia library for functional testing.](https://github.com/MechanicalRabbit/NarrativeTest.jl)**
>
> Julia library for functional testing.

- Write a Markdown description of behavior and pull out tests

> **[GitHub - fjarri/Jute.jl: A Py.Test-like testing framework for Julia](https://github.com/fjarri/Jute.jl)**
>
> A Py.Test-like testing framework for Julia

- Identifies tests, like functions, to collect them and then run.
- like py.test

> **[GitHub - IainNZ/Pukeko.jl: Testing for Julia, simplified.](https://github.com/IainNZ/Pukeko.jl)**
>
> Testing for Julia, simplified.

- Puts tests in functions in modules

> **[GitHub - tkf/LiterateTest.jl](https://github.com/tkf/LiterateTest.jl)**
>
> Contribute to tkf/LiterateTest.jl development by creating an account on GitHub.

- Literate programming tools (like Sweave) to write tests

# Testing tools

## Performance testing

> **[GitHub - JuliaTesting/PerformanceTestTools.jl](https://github.com/JuliaTesting/PerformanceTestTools.jl)**
>
> Contribute to JuliaTesting/PerformanceTestTools.jl development by creating an account on GitHub.

- enables nonfunctional performance testing inside CI tools
- adds an include macro that runs tests in different process

## Regression testing

> **[GitHub - JuliaTesting/ReferenceTests.jl: Utility package for comparing data against...](https://github.com/JuliaTesting/ReferenceTests.jl)**
>
> Utility package for comparing data against reference files

- compares values with values in files

> **[GitHub - maleadt/creduce\_julia](https://github.com/maleadt/creduce_julia)**
>
> Contribute to maleadt/creduce\_julia development by creating an account on GitHub.

- Generates smaller testing examples.
- Reduce size of suite while keeping behavior

## Floating-point assessment

> **[GitHub - JuliaIntervals/IntervalArithmetic.jl: Library for validated numerics using interval...](https://github.com/JuliaIntervals/IntervalArithmetic.jl)**
>
> Library for validated numerics using interval arithmetic

- Run a function on an interval in order to find numerical accuracy problems.

BigFloat can set rounding mode, which is a common heuristic to  
understand sensitivity of a calculation to numerical problems.

## Mocking

> **[GitHub - JuliaTesting/Mocking.jl: Allows Julia function calls to be temporarily...](https://github.com/JuliaTesting/Mocking.jl)**
>
> Allows Julia function calls to be temporarily overloaded for the purpose of testing

- mock a function

> **[GitHub - JuliaTesting/SimpleMock.jl: A basic mocking module](https://github.com/JuliaTesting/SimpleMock.jl)**
>
> A basic mocking module

- Mocking as a do-context

> **[GitHub - tk3369/Pretend.jl: Let's play pretend!](https://github.com/tk3369/Pretend.jl)**
>
> Let's play pretend!

- Mocking by annotating a function that you want to patch later

## Mutation testing

> **[GitHub - MikeInnes/Vimes.jl: Mutation Testing for Julia](https://github.com/MikeInnes/Vimes.jl)**
>
> Mutation Testing for Julia

- Mutation testing

> **[GitHub - vchuravy/ConcolicFuzzer.jl: Prototype of a fuzzer for Julia based on concolic...](https://github.com/vchuravy/ConcolicFuzzer.jl)**
>
> Prototype of a fuzzer for Julia based on concolic execution

- Fuzzer using concolic (concrete + symbolic) execution
- Runs code, watches it run, uses that to make the next choice.

## Generation of parameter values

> **[GitHub - eschnett/Arbitrary.jl: Generate arbitrary sequences for testing](https://github.com/eschnett/Arbitrary.jl)**
>
> Generate arbitrary sequences for testing

- Generates parameter values from the input domain
- Weights random generation to emphasize edge cases
- Like Python’s Hypothesis

[https://gitlab.com/quf/randomizedpropertytest.jl](https://gitlab.com/quf/randomizedpropertytest.jl)

- Randomized property testing
- QuickCheck inspired

> **[GitHub - adolgert/UnitTestDesign.jl: Automated unit test coverage with fewer test cases...](https://github.com/adolgert/UnitTestDesign.jl)**
>
> Automated unit test coverage with fewer test cases using all-pairs and other covering arrays.

- Factorial test designs
- All-pairs, all-triples.

# Code Coverage

> **[GitHub - robertfeldt/CodeTracer.jl: Instrument Julia code so we can trace it,...](https://github.com/robertfeldt/CodeTracer.jl)**
>
> Instrument Julia code so we can trace it, calculate coverage, do mutation testing etc...

- Instrument code to get coverage

> **[GitHub - bcbi/MicroCoverage.jl: Code coverage generation for Julia, implemented in...](https://github.com/bcbi/MicroCoverage.jl)**
>
> Code coverage generation for Julia, implemented in pure Julia

- Instruments for smaller code coverage by line

# Test specific tools

> **[GitHub - JuliaDebug/TerminalRegressionTests.jl: Regression Testing for Terminal UIs](https://github.com/JuliaDebug/TerminalRegressionTests.jl)**
>
> Regression Testing for Terminal UIs

- Regression testing of user interfaces to command-line sessions

> **[GitHub - JuliaLinearAlgebra/MatrixDepot.jl: An Extensible Test Matrix Collection for Julia](https://github.com/JuliaLinearAlgebra/MatrixDepot.jl)**
>
> An Extensible Test Matrix Collection for Julia

- A suite of matrices used for testing algorithms

> **[GitHub - JuliaHomotopyContinuation/PolynomialTestSystems.jl: A collection of systems of polynomials for...](https://github.com/JuliaHomotopyContinuation/PolynomialTestSystems.jl)**
>
> A collection of systems of polynomials for benchmarking, testing etc.

- Polynomials to solve for benchmarking

> **[GitHub - beacon-biosignals/ElectronTests.jl: Common utilities for testing JSServe apps with...](https://github.com/beacon-biosignals/ElectronTests.jl)**
>
> Common utilities for testing JSServe apps with Electron

- run javascript to test within Electron

> **[GitHub - JuliaDiff/DiffTests.jl: A common suite of test functions for stressing the...](https://github.com/JuliaDiff/DiffTests.jl)**
>
> A common suite of test functions for stressing the robustness of differentiation tools.

- Specific to testing differentiation tools

> **[GitHub - JuliaImages/TestImages.jl: commonly used test images](https://github.com/JuliaImages/TestImages.jl)**
>
> commonly used test images

- Images for testing

> **[GitHub - JuliaTesting/BrokenRecord.jl: Reproducible tests for HTTP requests](https://github.com/JuliaTesting/BrokenRecord.jl)**
>
> Reproducible tests for HTTP requests

- Record and replay http tests

> **[GitHub - JuliaPlots/VisualRegressionTests.jl: Automated integrated regression tests for graphics...](https://github.com/JuliaPlots/VisualRegressionTests.jl)**
>
> Automated integrated regression tests for graphics libraries

- Compare images for testing

> **[GitHub - JuliaNLSolvers/OptimTestProblems.jl: A collection of test problems for optimization...](https://github.com/JuliaNLSolvers/OptimTestProblems.jl)**
>
> A collection of test problems for optimization problems.

- Tests of optimization problems

> **[GitHub - JuliaDiff/ChainRulesTestUtils.jl: Utilities for testing custom AD primitives.](https://github.com/JuliaDiff/ChainRulesTestUtils.jl)**
>
> Utilities for testing custom AD primitives.

- Test automatic differentiation primitives

> **[GitHub - JuliaRandom/RNGTest.jl: Code for testing of Julia's random numbers](https://github.com/JuliaRandom/RNGTest.jl)**
>
> Code for testing of Julia's random numbers

- Test of random number generation

> **[GitHub - JuliaPOMDP/POMDPModels.jl: Problem models for testing POMDPs.jl](https://github.com/JuliaPOMDP/POMDPModels.jl)**
>
> Problem models for testing POMDPs.jl

- Test of POMDP models

> **[GitHub - tpapp/NonlinearTestProblems.jl: A Julia package collecting \`ℝⁿ → ℝᵐ\` functions...](https://github.com/tpapp/NonlinearTestProblems.jl)**
>
> A Julia package collecting \`ℝⁿ → ℝᵐ\` functions commonly used for testing nonlinear solvers and optimizers.

- Functions to test nonlinear optimization

> **[GitHub - jump-dev/MINLPTests.jl: Unit and Integration Tests for JuMP NLP and MINLP...](https://github.com/jump-dev/MINLPTests.jl)**
>
> Unit and Integration Tests for JuMP NLP and MINLP solvers

- Tests for nonlinear / polynomial solvers.

> **[GitHub - JuliaPhylo/QuartetNetworkGoodnessFit.jl: julia package for phylogenetic networks analyses...](https://github.com/JuliaPhylo/QuartetNetworkGoodnessFit.jl)**
>
> julia package for phylogenetic networks analyses using 4-taxon subsets.

- Data for tests of phylogenetic networks

> **[GitHub - JuliaTelecom/PhysicalCommunications.jl: Tools for development & test of PHY communication...](https://github.com/JuliaTelecom/PhysicalCommunications.jl)**
>
> Tools for development & test of PHY communication layer

- Tests of the physical layer (OSI model)

> **[GitHub - rbalexander1/BenchmarkFunctions.jl: A set of common benchmark functions for testing...](https://github.com/rbalexander1/BenchmarkFunctions.jl)**
>
> A set of common benchmark functions for testing optimization algorithms in Julia

- Functions to optimize to test optimization

> **[GitHub - BioJulia/FormatSpecimens.jl: Biological file format specimens for testing...](https://github.com/BioJulia/FormatSpecimens.jl)**
>
> Biological file format specimens for testing packages, and functions to assist in testing.

- Biological file format specimens

# Code rewriting

> **[GitHub - JuliaLabs/Cassette.jl: Overdub Your Julia Code](https://github.com/JuliaLabs/Cassette.jl)**
>
> Overdub Your Julia Code

- Insert behavior into the JIT

> **[GitHub - FluxML/MacroTools.jl: MacroTools provides a library of tools for working...](https://github.com/FluxML/MacroTools.jl)**
>
> MacroTools provides a library of tools for working with Julia code and expressions.

- Work with Julia code and expressions

> **[GitHub - chakravala/Reduce.jl: Symbolic parser for Julia language term rewriting...](https://github.com/chakravala/Reduce.jl)**
>
> Symbolic parser for Julia language term rewriting using REDUCE algebra

- Symbolic parser generator

> **[GitHub - thautwarm/MLStyle.jl: Julia functional programming infrastructures and...](https://github.com/thautwarm/MLStyle.jl)**
>
> Julia functional programming infrastructures and metaprogramming facilities

- Pattern-matching metaprogramming

> **[GitHub - dfdx/Espresso.jl: Expression transformation package](https://github.com/dfdx/Espresso.jl)**
>
> Expression transformation package

- Tools to rewrite the AST

[https://github.com/FluxML/IRTools.jl](https://github.com/FluxML/IRTools.jl)

- Provides an intermediate representation to work with.
- Works with Cassette.jl