Defining tests next to functions

For languages without classes tests are in my opinion a great way to document functions. For example, when I have the function

function str2int(letter::String) 
	letter = uppercase(letter)
	@assert occursin(r"[A-Z]+", letter)
	ch = letter[1]
	i = Int(ch) - 64

then the goal is immediately clear when I read

@test str2int("C") == 3

Unfortunately, all languages I know of tend to put these test in a file far away from the definition of the actual function. When we define these tests right below these functions then any time we load the module all tests will be run, which is not useful in practice either. To solve this I wrote a little macro (thanks to the Julia language for allowing that).

module DTest # `Delayed test`, or `define test`, whatever you like

all_dtests = Expr[]
export all_dtests

macro dtests(ex)
	push!(all_dtests, ex)
	# Returning last evaluated expression to get "Test Passed".
	esc(:(dtest() = eval.(all_dtests)[end]))
export @dtests

end # module

Then to use it I write

using Test
using .DTest

function str2int(letter::String) 
	letter = uppercase(letter)
	@assert occursin(r"[A-Z]+", letter)
	ch = letter[1]
	i = Int(ch) - 64
export str2int
@dtests begin
	@test str2int("C") == 3

Now I can still load the module Helpers and its functions without running the tests. To run the tests I call Helpers.dtest().

So far the small macro has been very convenient for me. I am wondering whether it is something I should make a Julia package, or even whether it is a nice addition to the standard library?


Neat, it is similar to:

BTW, I think that because of the way precompilation works, @dtest wouldn’t work in other packages, because the push! wouldn’t happen inside of __init__


Without thinking about the technical implications or problems with implementation of such a package, I like the idea and reading your post immediately convinced me that this is a good idea.

Not so much for the users of code, but for the subsequent developers who try to add functionality or fixing bugs.

1 Like

There are many ways to solve the problem. One comment line telling in which file the tests are located is one option.

One can follow a naming convention for this, eg tests for src/something.jl go in test/test_something.jl, in the same order.


What about doctests?
Wouldn’t that essentially do what you suggested?


Tamas_Papp Tero_Frondelius
Both your suggestions are true. However the reason I came up with this is out of convenience. Reminds me of the Hacker News comment where someone claimed that Dropbox was useless, because it can easily be created by just combining some Linux tools.

@cstjean @PetrKryslUCSD
Awesome. Did not know that existed. I think it would solve my problems indeed! Thanks

1 Like

Are you sure about that ? My understanding was that they are run only on precompilation, and that only the __init__() function is run when using the package.

1 Like

Fair. This depends on how you would define “load”. I’ve checked and indeed the tests trigger upon include("some_module.jl") and not on using .SomeModule.

Anyway we can mark this topic as solved. I asked whether it would be interesting and apparently we can use doctests.

Personally, Documenter has always kind of intimidated me and it seemed like far too much work to wade through it’s documentation to learn how to use it. It’d be nice if the doctest functionality could be separated out from the rest of the package and made to work on individual functions.

I’m imagining something like

julia> begin

       add `1` to `x`
       julia> f(1)
       f(x) = x + 1

julia> @doctest f
┌ Info: Testing the documentation for f: total 1 test
└ Test passed!

instead of requiring all that formal structure required by documenter. You could then integrate it into your test-suite like so:

using Test

@test @doctest(f, quiet)

I think it is a very nice package which is a key part of the ecosystem, and well worth the investment.

Perhaps one of the package template generators will make it easier to get started, eg


Not for individual functions, but you can run doctests in a package using Documenter without setting up docs/make.jl now:

using MyPackage
using Documenter: doctest
doctest(MyPackage; manual = false)



Another location I like for test files is in ‘src’ next to the file under test, e.g. ‘src/something.jl’ and ‘src/something_test.jl’. Tradeoff being less navigation between src and tests, but still less “clutter” navigating source code than if tests were dominating those files.

Pretty simple to set up in ‘runtests.jl’:

1 Like

With a reasonable editor, quick navigation between files in various directories and their buffers should not be an issue.

While one can of course always deviate from the standard file layout of Julia packages, I am not sure this should be done without a good reason, as it makes cooperation more difficult.

Seems to me that depends on the user and a bit on how source is organized. I think Vim and VSCode plus plugins are considered reasonable, maybe it’s the user who’s lacking here.

My understanding is that the standard specifies that there should be a ‘test/runtests.jl’, not where each test file should be, which I see varying across the ecosystem even when within ‘test’. That’s not to say I go off-script in projects I think could end up in General, but in terms of ‘making cooperation more difficult’ the option I presented here is about as innocuous as it gets – it seems it has immediate comprehension benefits for some, and couldn’t be as hard to adapt to for anyone as, say, what subset of the language to use, or what ‘quality of life’ dependencies to take on.

I.e. seems reasonable to not like it yourself, but it may be convenient to others and I don’t buy the ‘barrier to collaboration’ claim in this particular instance, though I may be missing something.

I would love to see a “delayed testing” type of module, where some lazy tests and examples can be defined, but not yet executed, right next to method definitions. I also feel like this is well suited for a macro (rather than a docstring).

In C++ I use GitHub - onqtam/doctest: The fastest feature-rich C++11/14/17/20 single-header testing framework exactly for this purpose

This looks like a good alternative: