Validity of package documentation and examples

General Usage
1

Is this really right?

Julia packages are rather messy and instructions and exampels for using packages seems not quite able to be in step with modifications.

2

That has been my experience as well. Many packages don’t seem to test their examples using the doctest functionality of Documenter.jl, which means they are often incorrect. I wonder how we can encourage doctest usage.

1 Like
3

I think this deserves a new thread, but for me the main difficulty is that the jldoctests cannot be run independently of the generation of the docs (AFAIK). It would be nice if there was a macro like

@jldoctest myfunction

that just ran the test and returned true or false. That could even aid just incorporating the jldoctests into the test set of the package.

3 Likes
4

My experience is that often not enough examples are given. More examples would (hopefully) also lead to more doctest usage :wink:

1 Like
5

Yep, they can! See Doctests · Documenter.jl

You can even setup a GitHub actions workflow to provide suggestions to fix them, inline in the PR: https://github.com/beacon-biosignals/KeywordSearch.jl/blob/main/.github/workflows/doctests.yml. I think it makes for a great workflow. (It is not widely known afaik though)

8 Likes
6

That looks useful. Could the regular test runner function be changed to automatically run doctests by default? I think doctests need to execute automatically for them to be effective.

7

If you look at it from the other hand, it’s a great opportunity to get some nice PRs in. Generally, repository owners which already have some CI are very happy to add doctests.

3 Likes
8

For example:

The second one also runs the doctests on the README.md file

9

I don’t disagree that documentation and documentation drift are often sore points, but seeing as messiness is a relative measure, are there Julia packages or other language ecosystems you think do it well? It would be great to have some positive examples we can look at :slight_smile:

3 Likes
10

The most popular Python test runner Pytest supports doctests natively – no separate make command is needed, and doctest has been in the standard library since early days.

I think testing docs automatically is a practical way to do it; otherwise people forget to run them or don’t see that their examples are broken.

  • For projects that build docs, the building process could run doctests, including the readme.
  • For projects whose docs are just in the readme, the testing process could test the doctests.

As for the writing of doctests, it might be useful to apply the perspective from code test-coverage:

  • what fraction of the package’s methods provided are documented with examples?
  • what fraction of the branches in the documentation code examples are executed in doctests?
11

I have seen the docs (and internals :wink:) of quite a few Julia packages, and I would not make broad statements like this. Some are messy, but a lot of them, especially the mature ones, are well-organized and documented.

In any case, the best approach is constructive: if you see something that can be improved, just open an issue or make a PR.

8 Likes
12

I think the code examples in docs and readmes are so often incorrect that it’s more of a systemic problem than an individual-package problem. This is a good opportunity to discuss systemic solutions. I tried to give some thoughts above but maybe other people can contribute too.

One issue is that there are a lot of code blocks that use

```julia

rather than

```jldoctest

or @example even though they are self-contained examples. I’m not sure why this is happening.

13

Worth stressing that you do not have to be building docs to do this. You do not have to set up a separate github action or whatever. In the ordinary runtests.jl, you can test all docstrings in your package, as an ordinary testset; they will run when testing locally, as well as on the simplest possible CI.

(The docs linked above are clear on this.)

And since docstrings are right next to the function you’re writing, they are easier to write than actual documentation, easier to update while in progress.

there are a lot of code blocks that use ```julia

Indeed. Sometimes these contain random numbers, which would make them fail as doctests. Maybe Documenter should have an option to check that these run without errors, without demanding that the outputs also match?

3 Likes
14

That might work; I don’t have good intuition about whether people would use it or not.

Could doctest() automatically set a fixed seed that before running each doctest?

Could doctest be changed to execute julia blocks unless marked they are marked julia nodoctest?

15

Also some, particularly small, packages, are documented directly in the README.md file of the github repo. There the examples are not run (AFAIK) and, additionally, they do not get properly syntax highlighted if using jldoctest. Is there anything that can be done about that?

1 Like
16

You can do those, so called, smoke tests by ending the lines with semicolons to suppress the output.

When the called methods print output, then that’s a bit more tricky, but also possible.

See also: https://github.com/JuliaDocs/Documenter.jl/issues/452

17

See my example above! Both problems are handled in the EffectSizes PR that I linked.

I’m also trying to add it to the Documenter.jl docs because I find outdated READMEs very annoying: https://github.com/JuliaDocs/Documenter.jl/pull/1664

2 Likes
18

That means the documentation doesn’t include the output which is less helpful.

19

What if my package doesn’t have an actual Documenter-type documentation, only a README.md? I would like to test the examples of the README.md and ones of the function comments. Is there a way
to do that, without having to copy/paste, which is prone to errors and cannot be scripted? (as nov/2020 it appears there wasn’t one)

(something that did not get clear to me: if I use jldoctest in the README.md file, it will appear properly syntax highlighted in github, I mean, after some of these PRs?).

20

In the project, add:

module MyModule

function _update_module_doc()
    path = joinpath(@__DIR__, "..", "README.md")
    text = read(path, String)
    # The code blocks in the README.md should be julia blocks for the syntax highlighter.
    text = replace(text, "```julia" => "```jldoctest")
    @doc text MyModule
end
_update_module_doc()

end # module

Then, in runtests.jl add:

using Documenter

DocMeta.setdocmeta!(
    MyModule,
    :DocTestSetup,
    :(using MyModule);
    recursive=true
)
MyModule._update_module_doc()
doctest(MyModule)

and in the README.md add:

 ```julia
 julia> true
 false
 ```

Now, the README.md gets proper syntax highlighting, because it is a julia code block, and the doctest runs when doing Pkg.test("MyModule"). Specifically, in this example the doctests will fail because true is not false.

That’s fine. The EffectSizes project linked above also doesn’t have documentation. When not using Documenter for building docs, it is also necessary to add a file in src/docs. This file can be empty; it’s fine as long as there is something.

1 Like