# Defining tests next to functions

**URL:** <https://discourse.julialang.org/t/defining-tests-next-to-functions/32155>\
**Category:** Internals & Design\
**Tags:** proposal, testing\
**Created:** [December 11, 2019, 3:19pm UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155 "2019-12-11T15:19:51Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [December 11, 2019, 3:19pm UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/1 "2019-12-11T15:19:51Z")

</div>

For languages without classes tests are in my opinion a great way to document functions. For example, when I have the function

```julia
function str2int(letter::String) 
	letter = uppercase(letter)
	@assert occursin(r"[A-Z]+", letter)
	ch = letter[1]
	i = Int(ch) - 64
end

```

then the goal is immediately clear when I read

```julia
@test str2int("C") == 3

```

Unfortunately, all languages I know of tend to put these test in a file far away from the definition of the actual function. When we define these tests right below these functions then any time we load the module all tests will be run, which is not useful in practice either. To solve this I wrote a little macro (thanks to the Julia language for allowing that).

```julia
module DTest # `Delayed test`, or `define test`, whatever you like

all_dtests = Expr[]
export all_dtests

macro dtests(ex)
	push!(all_dtests, ex)
	# Returning last evaluated expression to get "Test Passed".
	esc(:(dtest() = eval.(all_dtests)[end]))
end
export @dtests

end # module

```

Then to use it I write

```julia
using Test
include("dtest.jl")
using .DTest

function str2int(letter::String) 
	letter = uppercase(letter)
	@assert occursin(r"[A-Z]+", letter)
	ch = letter[1]
	i = Int(ch) - 64
end
export str2int
@dtests begin
	@test str2int("C") == 3
end

```

Now I can still load the module `Helpers` and its functions **without** running the tests. To run the tests I call `Helpers.dtest()`.

So far the small macro has been very convenient for me. I am wondering whether it is something I should make a Julia package, or even whether it is a nice addition to the standard library?

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [December 11, 2019, 3:29pm UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/2 "2019-12-11T15:29:33Z")

</div>

Neat, it is similar to:

[https://juliadocs.github.io/Documenter.jl/stable/man/doctests/index.html](https://juliadocs.github.io/Documenter.jl/stable/man/doctests/index.html)

BTW, I think that because of the way precompilation works, `@dtest` wouldn’t work in other packages, because the `push!` wouldn’t happen inside of ` __init__ `

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [December 11, 2019, 3:34pm UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/3 "2019-12-11T15:34:15Z")

</div>

Without thinking about the technical implications or problems with implementation of such a package, I like the idea and reading your post immediately convinced me that this is a good idea.

Not so much for the users of code, but for the subsequent developers who try to add functionality or fixing bugs.

---

<div class="post-metadata">

**Author:** ![Tero\_Frondelius](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tero_frondelius/32/7629_2.png) [@Tero\_Frondelius](https://discourse.julialang.org/u/Tero_Frondelius)\
**Post date:** [December 11, 2019, 4:33pm UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/4 "2019-12-11T16:33:07Z")

</div>

There are many ways to solve the problem. One comment line telling in which file the tests are located is one option.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 12, 2019, 9:15am UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/5 "2019-12-12T09:15:05Z")

</div>

One can follow a naming convention for this, eg tests for `src/something.jl` go in `test/test_something.jl`, in the same order.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [December 12, 2019, 9:19am UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/6 "2019-12-12T09:19:23Z")

</div>

What about doctests? [https://docs.julialang.org/en/latest/manual/documentation/#Documentation-1](https://docs.julialang.org/en/latest/manual/documentation/#Documentation-1)  
Wouldn’t that essentially do what you suggested?

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [December 12, 2019, 9:34am UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/7 "2019-12-12T09:34:47Z")

</div>

Tamas\_Papp Tero\_Frondelius  
Both your suggestions are true. However the reason I came up with this is out of convenience. Reminds me of the Hacker News comment where someone claimed that Dropbox was useless, because it can easily be created by just combining some Linux tools.

@cstjean @PetrKryslUCSD  
Awesome. Did not know that existed. I think it would solve my problems indeed! Thanks

---

<div class="post-metadata">

**Author:** ![jonathanBieler](https://avatars.discourse-cdn.com/v4/letter/j/82dd89/32.png) [@jonathanBieler](https://discourse.julialang.org/u/jonathanBieler)\
**Post date:** [December 12, 2019, 1:40pm UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/8 "2019-12-12T13:40:16Z")

</div>

> [@rikh](#):
>
> When we define these tests right below these functions then any time we load the module all tests will be run, which is not useful in practice either.

Are you sure about that ? My understanding was that they are run only on precompilation, and that only the ` __init__ ()` function is run when using the package.

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [December 12, 2019, 3:33pm UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/9 "2019-12-12T15:33:32Z")

</div>

Fair. This depends on how you would define “load”. I’ve checked and indeed the tests trigger upon `include("some_module.jl")` and not on `using .SomeModule`.

Anyway we can mark this topic as solved. I asked whether it would be interesting and apparently we can use doctests.

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [December 12, 2019, 7:10pm UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/10 "2019-12-12T19:10:15Z")

</div>

Personally, Documenter has always kind of intimidated me and it seemed like far too much work to wade through it’s documentation to learn how to use it. It’d be nice if the doctest functionality could be separated out from the rest of the package and made to work on individual functions.

I’m imagining something like

````julia
julia> begin
       """
          f(x)

       add `1` to `x`
       ```jldoctest
       julia> f(1)
       2
       ```
       """
       f(x) = x + 1
       end
f

julia> @doctest f
┌ Info: Testing the documentation for f: total 1 test
└ Test passed!
true

````

instead of requiring all that formal structure required by documenter. You could then integrate it into your test-suite like so:

```julia
using Test

@test @doctest(f, quiet)

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 13, 2019, 8:53am UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/11 "2019-12-13T08:53:21Z")

</div>

> [@Mason](#):
>
> Documenter has always kind of intimidated me and it seemed like far too much work to wade through it’s documentation to learn how to use it.

I think it is a very nice package which is a key part of the ecosystem, and well worth the investment.

Perhaps one of the package template generators will make it easier to get started, eg

> **[GitHub - JuliaCI/PkgTemplates.jl: Create new Julia packages, the easy way](https://github.com/JuliaCI/PkgTemplates.jl)**
>
> Create new Julia packages, the easy way. Contribute to JuliaCI/PkgTemplates.jl development by creating an account on GitHub.

> **[GitHub - tpapp/PkgSkeleton.jl: Generate Julia package skeletons using a...](https://github.com/tpapp/PkgSkeleton.jl)**
>
> Generate Julia package skeletons using a simple template system - GitHub - tpapp/PkgSkeleton.jl: Generate Julia package skeletons using a simple template system

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [December 13, 2019, 10:04am UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/12 "2019-12-13T10:04:19Z")

</div>

> [@Mason](#):
>
> It’d be nice if the doctest functionality could be separated out from the rest of the package and made to work on individual functions.

Not for individual functions, but you can run doctests in a package using Documenter without setting up `docs/make.jl` now:

```julia
using MyPackage
using Documenter: doctest
doctest(MyPackage; manual = false)

```

See: [Doctests · Documenter.jl](https://juliadocs.github.io/Documenter.jl/dev/man/doctests/#Doctesting-as-Part-of-Testing-1)

---

<div class="post-metadata">

**Author:** ![garborg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/garborg/32/11446_2.png) [@garborg](https://discourse.julialang.org/u/garborg)\
**Post date:** [December 13, 2019, 12:54pm UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/13 "2019-12-13T12:54:10Z")

</div>

Another location I like for test files is in ‘src’ next to the file under test, e.g. ‘src/something.jl’ and ‘src/something\_test.jl’. Tradeoff being less navigation between src and tests, but still less “clutter” navigating source code than if tests were dominating those files.

Pretty simple to set up in ‘runtests.jl’: [https://github.com/garborg/MyPkgTemplates.jl/blob/25ee1139338e44dded7989f381203da7c1379526/src/plugin/TestsInSrc.jl#L15-L23](https://github.com/garborg/MyPkgTemplates.jl/blob/25ee1139338e44dded7989f381203da7c1379526/src/plugin/TestsInSrc.jl#L15-L23)

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 13, 2019, 1:27pm UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/14 "2019-12-13T13:27:43Z")

</div>

> [@garborg](#):
>
> Tradeoff being less navigation between src and tests

With a reasonable editor, quick navigation between files in various directories and their buffers should not be an issue.

While one can of course always deviate from the standard file layout of Julia packages, I am not sure this should be done without a good reason, as it makes cooperation more difficult.

---

<div class="post-metadata">

**Author:** ![garborg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/garborg/32/11446_2.png) [@garborg](https://discourse.julialang.org/u/garborg)\
**Post date:** [December 13, 2019, 1:50pm UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/15 "2019-12-13T13:50:01Z")

</div>

> [@Tamas\_Papp](#):
>
> With a reasonable editor, quick navigation between files in various directories and their buffers should not be an issue.

Seems to me that depends on the user and a bit on how source is organized. I think Vim and VSCode plus plugins are considered reasonable, maybe it’s the user who’s lacking here.

> [@Tamas\_Papp](#):
>
> While one can of course always deviate from the standard file layout of Julia packages, I am not sure this should be done without a good reason, as it makes cooperation more difficult.

My understanding is that the standard specifies that there should be a ‘test/runtests.jl’, not where each test file should be, which I see varying across the ecosystem even when within ‘test’. That’s not to say I go off-script in projects I think could end up in General, but in terms of ‘making cooperation more difficult’ the option I presented here is about as innocuous as it gets – it seems it has immediate comprehension benefits for some, and couldn’t be as hard to adapt to for anyone as, say, what subset of the language to use, or what ‘quality of life’ dependencies to take on.

I.e. seems reasonable to not like it yourself, but it may be convenient to others and I don’t buy the ‘barrier to collaboration’ claim in this particular instance, though I may be missing something.

---

<div class="post-metadata">

**Author:** ![heetbeet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/heetbeet/32/20958_2.png) [@heetbeet](https://discourse.julialang.org/u/heetbeet)\
**Post date:** [February 1, 2021, 8:25am UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/16 "2021-02-01T08:25:36Z")

</div>

I would love to see a “delayed testing” type of module, where some lazy tests and examples can be defined, but not yet executed, right next to method definitions. I also feel like this is well suited for a macro (rather than a docstring).

---

<div class="post-metadata">

**Author:** ![heetbeet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/heetbeet/32/20958_2.png) [@heetbeet](https://discourse.julialang.org/u/heetbeet)\
**Post date:** [February 1, 2021, 8:26am UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/17 "2021-02-01T08:26:36Z")

</div>

In C++ I use [GitHub - doctest/doctest: The fastest feature-rich C++11/14/17/20 single-header testing framework](https://github.com/onqtam/doctest) exactly for this purpose

---

<div class="post-metadata">

**Author:** ![heetbeet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/heetbeet/32/20958_2.png) [@heetbeet](https://discourse.julialang.org/u/heetbeet)\
**Post date:** [February 19, 2021, 8:37am UTC](https://discourse.julialang.org/t/defining-tests-next-to-functions/32155/18 "2021-02-19T08:37:02Z")

</div>

This looks like a good alternative:

> **[GitHub - JuliaTesting/ReTest.jl: Testing framework for Julia](https://github.com/JuliaTesting/ReTest.jl)**
>
> Testing framework for Julia. Contribute to JuliaTesting/ReTest.jl development by creating an account on GitHub.