# Validity of package documentation and examples
**URL:** https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217
**Category:** General Usage
**Created:** [August 11, 2021, 6:47pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217 "2021-08-11T18:47:51Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@JOST](https://discourse.julialang.org/u/JOST)
#### Post date: [August 11, 2021, 6:47pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/1 "2021-08-11T18:47:51Z")
Is this really right?
> [@What is the advantage of Julia over Fortran?](https://discourse.julialang.org/t/what-is-the-advantage-of-julia-over-fortran/65964/1):
>
> It also have many packages which are easy to install and use.
Julia packages are rather messy and instructions and exampels for using packages seems not quite able to be in step with modifications.
---
### Author:  [@jzr](https://discourse.julialang.org/u/jzr)
#### Post date: [August 11, 2021, 6:59pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/2 "2021-08-11T18:59:12Z")
> [@JOST](#):
>
> instructions and exampels for using packages seems not quite able to be in step with modifications.
That has been my experience as well. Many packages don’t seem to test their examples using the [doctest functionality](https://juliadocs.github.io/Documenter.jl/stable/man/doctests/) of Documenter.jl, which means they are often incorrect. I wonder how we can encourage doctest usage.
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [August 11, 2021, 7:04pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/3 "2021-08-11T19:04:55Z")
> [@jzr](#):
>
> That has been my experience as well. Many packages don’t seem to test their examples using the [doctest functionality](https://juliadocs.github.io/Documenter.jl/stable/man/doctests/) of Documenter.jl, which means they are often incorrect. I wonder how we can encourage doctest usage.
I think this deserves a new thread, but for me the main difficulty is that the jldoctests cannot be run independently of the generation of the docs (AFAIK). It would be nice if there was a macro like
```julia
@jldoctest myfunction
```
that just ran the test and returned true or false. That could even aid just incorporating the jldoctests into the test set of the package.
---
### Author:  [@MatFi](https://discourse.julialang.org/u/MatFi)
#### Post date: [August 11, 2021, 7:08pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/4 "2021-08-11T19:08:19Z")
> [@jzr](#):
>
> Many packages don’t seem to test their examples using the [doctest functionality](https://juliadocs.github.io/Documenter.jl/stable/man/doctests/) of Documenter.jl, which means they are often incorrect. I wonder how we can encourage doctest usage.
My experience is that often not enough examples are given. More examples would (hopefully) also lead to more doctest usage 😉
---
### Author:  [@ericphanson](https://discourse.julialang.org/u/ericphanson)
#### Post date: [August 11, 2021, 7:40pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/5 "2021-08-11T19:40:27Z")
> [@lmiq](#):
>
> jldoctests cannot be run independently of the generation of the docs (AFAIK)
Yep, they can! See [Doctests · Documenter.jl](https://juliadocs.github.io/Documenter.jl/stable/man/doctests/#Doctesting-as-Part-of-Testing)
You can even setup a GitHub actions workflow to provide suggestions to fix them, inline in the PR: [https://github.com/beacon-biosignals/KeywordSearch.jl/blob/main/.github/workflows/doctests.yml](https://github.com/beacon-biosignals/KeywordSearch.jl/blob/main/.github/workflows/doctests.yml). I think it makes for a great workflow. (It is not widely known afaik though)
---
### Author:  [@jzr](https://discourse.julialang.org/u/jzr)
#### Post date: [August 11, 2021, 7:47pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/6 "2021-08-11T19:47:16Z")
> [@ericphanson](#):
>
> Yep, they can! See [Doctests · Documenter.jl](https://juliadocs.github.io/Documenter.jl/stable/man/doctests/#Doctesting-as-Part-of-Testing)
That looks useful. Could the regular test runner function be changed to automatically run doctests by default? I think doctests need to execute automatically for them to be effective.
---
### Author:  [@rikh](https://discourse.julialang.org/u/rikh)
#### Post date: [August 11, 2021, 8:42pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/7 "2021-08-11T20:42:17Z")
If you look at it from the other hand, it’s a great opportunity to get some nice PRs in. Generally, repository owners which already have some CI are very happy to add doctests.
---
### Author:  [@rikh](https://discourse.julialang.org/u/rikh)
#### Post date: [August 11, 2021, 8:46pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/8 "2021-08-11T20:46:11Z")
For example:
- [https://github.com/JuliaSymbolics/SymbolicUtils.jl/pull/333](https://github.com/JuliaSymbolics/SymbolicUtils.jl/pull/333)
- [https://github.com/harryscholes/EffectSizes.jl/pull/19](https://github.com/harryscholes/EffectSizes.jl/pull/19)
The second one also runs the doctests on the README.md file
---
### Author:  [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)
#### Post date: [August 11, 2021, 9:31pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/9 "2021-08-11T21:31:54Z")
I don’t disagree that documentation and documentation drift are often sore points, but seeing as messiness is a relative measure, are there Julia packages or other language ecosystems you think do it well? It would be great to have some positive examples we can look at 🙂
---
### Author:  [@jzr](https://discourse.julialang.org/u/jzr)
#### Post date: [August 11, 2021, 10:21pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/10 "2021-08-11T22:21:35Z")
> [@ToucheSir](#):
>
> other language ecosystems you think do it well?
The most popular Python test runner Pytest supports doctests natively – no separate `make` command is needed, and [`doctest`](https://docs.python.org/2.7/library/doctest.html) has been in the standard library since early days.
I think testing docs automatically is a practical way to do it; otherwise people forget to run them or don’t see that their examples are broken.
- For projects that build docs, the building process could run doctests, including the readme.
- For projects whose docs are just in the readme, the testing process could test the doctests.
As for the _writing_ of doctests, it might be useful to apply the perspective from code test-coverage:
- what fraction of the package’s methods provided are documented with examples?
- what fraction of the branches in the documentation code examples are executed in doctests?
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [August 13, 2021, 7:59am UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/11 "2021-08-13T07:59:21Z")
> [@JOST](#):
>
> Julia packages are rather messy
I have seen the docs (and internals 😉) of quite a few Julia packages, and I would not make broad statements like this. Some are messy, but a lot of them, especially the mature ones, are well-organized and documented.
In any case, the best approach is constructive: if you see something that can be improved, just open an issue or make a PR.
---
### Author:  [@jzr](https://discourse.julialang.org/u/jzr)
#### Post date: [August 16, 2021, 7:40pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/12 "2021-08-16T19:40:31Z")
I think the code examples in docs and readmes are so often incorrect that it’s more of a systemic problem than an individual-package problem. This is a good opportunity to discuss systemic solutions. I tried to give some thoughts above but maybe other people can contribute too.
One issue is that there are a lot of code blocks that use
````julia
```julia
````
rather than
````julia
```jldoctest
````
or `@example` even though they are self-contained examples. I’m not sure why this is happening.
---
### Author:  [@mcabbott](https://discourse.julialang.org/u/mcabbott)
#### Post date: [August 16, 2021, 8:16pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/13 "2021-08-16T20:16:41Z")
> [@jzr](#):
>
> For projects that build docs
Worth stressing that you do not have to be building docs to do this. You do not have to set up a separate github action or whatever. In the ordinary `runtests.jl`, you can test all docstrings in your package, as an ordinary testset; they will run when testing locally, as well as on the simplest possible CI.
(The docs linked above are clear on this.)
And since docstrings are right next to the function you’re writing, they are easier to write than actual documentation, easier to update while in progress.
> there are a lot of code blocks that use ```julia
Indeed. Sometimes these contain random numbers, which would make them fail as doctests. Maybe Documenter should have an option to check that these run without errors, without demanding that the outputs also match?
---
### Author:  [@jzr](https://discourse.julialang.org/u/jzr)
#### Post date: [August 16, 2021, 8:25pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/14 "2021-08-16T20:25:37Z")
> [@mcabbott](#):
>
> Sometimes these contain random numbers, which would make them fail as doctests. Maybe Documenter should have an option to check that these run without errors, without demanding that the outputs also match?
That might work; I don’t have good intuition about whether people would use it or not.
Could `doctest()` automatically set a fixed seed that before running each doctest?
Could doctest be changed to execute `julia` blocks unless marked they are marked `julia nodoctest`?
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [August 16, 2021, 8:29pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/15 "2021-08-16T20:29:22Z")
Also some, particularly small, packages, are documented directly in the README.md file of the github repo. There the examples are not run (AFAIK) and, additionally, they do not get properly syntax highlighted if using `jldoctest`. Is there anything that can be done about that?
---
### Author:  [@rikh](https://discourse.julialang.org/u/rikh)
#### Post date: [August 16, 2021, 9:46pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/16 "2021-08-16T21:46:27Z")
You can do those, so called, _smoke tests_ by ending the lines with semicolons to suppress the output.
When the called methods print output, then that’s a bit more tricky, but also possible.
See also: [https://github.com/JuliaDocs/Documenter.jl/issues/452](https://github.com/JuliaDocs/Documenter.jl/issues/452)
---
### Author:  [@rikh](https://discourse.julialang.org/u/rikh)
#### Post date: [August 16, 2021, 9:47pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/17 "2021-08-16T21:47:14Z")
See my example above! Both problems are handled in the EffectSizes PR that I linked.
I’m also trying to add it to the Documenter.jl docs because I find outdated READMEs very annoying: [https://github.com/JuliaDocs/Documenter.jl/pull/1664](https://github.com/JuliaDocs/Documenter.jl/pull/1664)
---
### Author:  [@jzr](https://discourse.julialang.org/u/jzr)
#### Post date: [August 16, 2021, 9:49pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/18 "2021-08-16T21:49:45Z")
> [@rikh](#):
>
> You can do those, so called, _smoke tests_ by ending the lines with semicolons to suppress the output.
That means the documentation doesn’t include the output which is less helpful.
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [August 18, 2021, 11:58am UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/19 "2021-08-18T11:58:08Z")
> [@rikh](#):
>
> See my example above! Both problems are handled in the EffectSizes PR that I linked.
What if my package doesn’t have an actual Documenter-type documentation, only a README.md? I would like to test the examples of the README.md and ones of the function comments. Is there a way
to do that, without having to copy/paste, which is prone to errors and cannot be scripted? ([as nov/2020 it appears there wasn’t one](https://discourse.julialang.org/t/is-there-a-way-to-test-individual-jldoctests/49930))
(something that did not get clear to me: if I use `jldoctest` in the README.md file, it will appear properly syntax highlighted in github, I mean, after some of these PRs?).
---
### Author:  [@rikh](https://discourse.julialang.org/u/rikh)
#### Post date: [August 18, 2021, 1:20pm UTC](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217/20 "2021-08-18T13:20:50Z")
In the project, add:
````julia
module MyModule
function _update_module_doc()
path = joinpath(@ __DIR__ , "..", "README.md")
text = read(path, String)
# The code blocks in the README.md should be julia blocks for the syntax highlighter.
text = replace(text, "```julia" => "```jldoctest")
@doc text MyModule
end
_update_module_doc()
end # module
````
Then, in `runtests.jl` add:
```julia
using Documenter
DocMeta.setdocmeta!(
MyModule,
:DocTestSetup,
:(using MyModule);
recursive=true
)
MyModule._update_module_doc()
doctest(MyModule)
```
and in the `README.md` add:
````
```julia
julia> true
false
```
````
Now, the `README.md` gets proper syntax highlighting, because it is a `julia` code block, and the doctest runs when doing `Pkg.test("MyModule")`. Specifically, in this example the doctests will fail because `true` is not `false`.
> [@lmiq](#):
>
> What if my package doesn’t have an actual Documenter-type documentation, only a README.md?
That’s fine. The EffectSizes project linked above also doesn’t have documentation. When not using Documenter for building docs, it is also necessary to add a file in `src/docs`. This file can be empty; it’s fine as long as there is something.
[Next page](https://discourse.julialang.org/t/validity-of-package-documentation-and-examples/66217.md?page=2)