Best practices for examples and CI

question
#1

I would like to include some worked examples in a package (distinct from unit tests).

  1. Where should they go? A subdirectory in docs (will that interfere with Documenter.jl?) or in tests?

  2. Is there a framework for running them within CI? For now, I just want to run each example on Travis in a fresh Julia process, and check that it completes without an error. Later on it would be nice to check the results.

  3. Is there a way I can splice a file into documentation generated with Documenter.jl as Julia source code, ie between ```s?

#2

It seems that #3 has an open issue already:

#3

You can use separate groups. See Turing.jl’s test setup:

Make the example in its own module and then add that module to the module list for Documenter?

#4

For my projects I decided to just use Jupyter notebooks. It is easier for the user to play with them right away, specially if you provide a dummy function to start Jupyter in the examples folder:

I keep updating them as the project evolves. The major downside of writing separate Jupyter notebooks is that they aren’t synced with the docs. If you update them in the repository, they will be always ahead of the latest tagged version.

#5

Also, you can use NBInclude.jl to run Jupyter notebooks on Travis, and make sure that the examples aren’t broken.

#6

In Query I have an examples folder, and run them as part of my tests. I’m not super happy with that because it really slows down the build, but at least I know when something breaks.

#7

Thanks for sharing this @cstjean, can be quite useful!

#8

I have somewhat of a hack at Augmentor.jl to convert plain examples/*.jl scripts into nice looking documentation pages as well as jupyter notebooks. Its more of a curiosity than a clean solution but it does its job quite nicely

example input script: https://github.com/Evizero/Augmentor.jl/blob/master/examples/mnist_tensorflow.jl
output doc page: https://evizero.github.io/Augmentor.jl/generated/mnist_tensorflow/
output notebook: https://nbviewer.jupyter.org/github/Evizero/Augmentor.jl/blob/gh-pages/generated/mnist_tensorflow.ipynb

#9

With a bit of rejiggering, you can test doctests as part of Pkg.test. You just have to move the code for building your documentation inside your testing file (set strict = TRUE and also some messing around with file paths). I do this for all of my packages, and try to have as many tests in the form as doctests as possible.

#10

Thanks! Can you link an example of testing standalone examples? I looked at some of your packages and saw how you build the docs in runtests.jl, but found no example testing.

#11

While I got many helpful answers for questions 2 and 3, no one said anything about 1, so I guess there is no standard location for examples. I will just use examples/.

#12

So for example NumberedLines has no proper tests, only doctests. Nevertheless coverage is 96%. https://bramtayl.github.io/NumberedLines.jl/stable/

#13

Maybe there should be a standard? I remember a time when we didn’t have a real standard for the src and test folders, now there is. Maybe a similar standardization for the examples would be good?

#14

As far as I can tell, examples/ seems like an unspoken standard