Defining tests next to functions

For languages without classes tests are in my opinion a great way to document functions. For example, when I have the function

function str2int(letter::String) 
	letter = uppercase(letter)
	@assert occursin(r"[A-Z]+", letter)
	ch = letter[1]
	i = Int(ch) - 64
end

then the goal is immediately clear when I read

@test str2int("C") == 3

Unfortunately, all languages I know of tend to put these test in a file far away from the definition of the actual function. When we define these tests right below these functions then any time we load the module all tests will be run, which is not useful in practice either. To solve this I wrote a little macro (thanks to the Julia language for allowing that).

module DTest # `Delayed test`, or `define test`, whatever you like

all_dtests = Expr[]
export all_dtests

macro dtests(ex)
	push!(all_dtests, ex)
	# Returning last evaluated expression to get "Test Passed".
	esc(:(dtest() = eval.(all_dtests)[end]))
end
export @dtests

end # module

Then to use it I write

using Test
include("dtest.jl")
using .DTest

function str2int(letter::String) 
	letter = uppercase(letter)
	@assert occursin(r"[A-Z]+", letter)
	ch = letter[1]
	i = Int(ch) - 64
end
export str2int
@dtests begin
	@test str2int("C") == 3
end

Now I can still load the module Helpers and its functions without running the tests. To run the tests I call Helpers.dtest().

So far the small macro has been very convenient for me. I am wondering whether it is something I should make a Julia package, or even whether it is a nice addition to the standard library?

6 Likes

Neat, it is similar to:

https://juliadocs.github.io/Documenter.jl/stable/man/doctests/index.html

BTW, I think that because of the way precompilation works, @dtest wouldn’t work in other packages, because the push! wouldn’t happen inside of __init__

2 Likes

Without thinking about the technical implications or problems with implementation of such a package, I like the idea and reading your post immediately convinced me that this is a good idea.

Not so much for the users of code, but for the subsequent developers who try to add functionality or fixing bugs.

There are many ways to solve the problem. One comment line telling in which file the tests are located is one option.

One can follow a naming convention for this, eg tests for src/something.jl go in test/test_something.jl, in the same order.

1 Like

What about doctests? https://docs.julialang.org/en/latest/manual/documentation/#Documentation-1
Wouldn’t that essentially do what you suggested?

4 Likes

Tamas_Papp Tero_Frondelius
Both your suggestions are true. However the reason I came up with this is out of convenience. Reminds me of the Hacker News comment where someone claimed that Dropbox was useless, because it can easily be created by just combining some Linux tools.

@cstjean @PetrKryslUCSD
Awesome. Did not know that existed. I think it would solve my problems indeed! Thanks

1 Like

Are you sure about that ? My understanding was that they are run only on precompilation, and that only the __init__() function is run when using the package.

1 Like

Fair. This depends on how you would define “load”. I’ve checked and indeed the tests trigger upon include("some_module.jl") and not on using .SomeModule.

Anyway we can mark this topic as solved. I asked whether it would be interesting and apparently we can use doctests.

Personally, Documenter has always kind of intimidated me and it seemed like far too much work to wade through it’s documentation to learn how to use it. It’d be nice if the doctest functionality could be separated out from the rest of the package and made to work on individual functions.

I’m imagining something like

julia> begin
       """
          f(x)

       add `1` to `x`
       ```jldoctest
       julia> f(1)
       2
       ```
       """
       f(x) = x + 1
       end
f

julia> @doctest f
┌ Info: Testing the documentation for f: total 1 test
└ Test passed!
true

instead of requiring all that formal structure required by documenter. You could then integrate it into your test-suite like so:

using Test

@test @doctest(f, quiet)
5 Likes

I think it is a very nice package which is a key part of the ecosystem, and well worth the investment.

Perhaps one of the package template generators will make it easier to get started, eg

2 Likes

Not for individual functions, but you can run doctests in a package using Documenter without setting up docs/make.jl now:

using MyPackage
using Documenter: doctest
doctest(MyPackage; manual = false)

See: https://juliadocs.github.io/Documenter.jl/dev/man/doctests/#Doctesting-as-Part-of-Testing-1

7 Likes

Another location I like for test files is in ‘src’ next to the file under test, e.g. ‘src/something.jl’ and ‘src/something_test.jl’. Tradeoff being less navigation between src and tests, but still less “clutter” navigating source code than if tests were dominating those files.

Pretty simple to set up in ‘runtests.jl’: https://github.com/garborg/MyPkgTemplates.jl/blob/25ee1139338e44dded7989f381203da7c1379526/src/plugin/TestsInSrc.jl#L15-L23

1 Like

With a reasonable editor, quick navigation between files in various directories and their buffers should not be an issue.

While one can of course always deviate from the standard file layout of Julia packages, I am not sure this should be done without a good reason, as it makes cooperation more difficult.

Seems to me that depends on the user and a bit on how source is organized. I think Vim and VSCode plus plugins are considered reasonable, maybe it’s the user who’s lacking here.

My understanding is that the standard specifies that there should be a ‘test/runtests.jl’, not where each test file should be, which I see varying across the ecosystem even when within ‘test’. That’s not to say I go off-script in projects I think could end up in General, but in terms of ‘making cooperation more difficult’ the option I presented here is about as innocuous as it gets – it seems it has immediate comprehension benefits for some, and couldn’t be as hard to adapt to for anyone as, say, what subset of the language to use, or what ‘quality of life’ dependencies to take on.

I.e. seems reasonable to not like it yourself, but it may be convenient to others and I don’t buy the ‘barrier to collaboration’ claim in this particular instance, though I may be missing something.