# Best practices for examples and CI

**URL:** <https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304>\
**Category:** General Usage\
**Tags:** question\
**Created:** [November 25, 2017, 12:55pm UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304 "2017-11-25T12:55:41Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 25, 2017, 12:55pm UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/1 "2017-11-25T12:55:41Z")

</div>

I would like to include some worked examples in a package (distinct from unit tests).

1. _Where_ should they go? A subdirectory in `docs` (will that interfere with `Documenter.jl`?) or in `tests`?

2. Is there a framework for _running them_ within CI? For now, I just want to run each example on Travis in a fresh Julia process, and check that it completes without an error. Later on it would be nice to check the results.

3. Is there a way I can _splice_ a file into documentation generated with `Documenter.jl` as Julia source code, ie between ```s?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 25, 2017, 2:06pm UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/2 "2017-11-25T14:06:27Z")

</div>

It seems that #3 has an open issue already:  
[https://github.com/JuliaDocs/Documenter.jl/issues/499](https://github.com/JuliaDocs/Documenter.jl/issues/499)

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 25, 2017, 2:08pm UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/3 "2017-11-25T14:08:19Z")

</div>

> [@Tamas\_Papp](#):
>
> Is there a framework for running them within CI? For now, I just want to run each example on Travis in a fresh Julia process, and check that it completes without an error. Later on it would be nice to check the results.

You can use separate groups. See Turing.jl’s test setup:

[https://github.com/yebai/Turing.jl/blob/master/.travis.yml](https://github.com/yebai/Turing.jl/blob/master/.travis.yml)

> [@Tamas\_Papp](#):
>
> Is there a way I can splice a file into documentation generated with Documenter.jl as Julia source code, ie between ```s?

Make the example in its own module and then add that module to the module list for Documenter?

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [November 26, 2017, 12:53am UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/4 "2017-11-26T00:53:53Z")

</div>

For my projects I decided to just use Jupyter notebooks. It is easier for the user to play with them right away, specially if you provide a dummy function to start Jupyter in the examples folder:

[https://github.com/juliohm/GeoStats.jl/blob/master/src/GeoStats.jl#L62-L66](https://github.com/juliohm/GeoStats.jl/blob/master/src/GeoStats.jl#L62-L66)

I keep updating them as the project evolves. The major downside of writing separate Jupyter notebooks is that they aren’t synced with the docs. If you update them in the repository, they will be always ahead of the latest tagged version.

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [November 26, 2017, 1:39am UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/5 "2017-11-26T01:39:32Z")

</div>

Also, you can use [NBInclude.jl](https://github.com/stevengj/NBInclude.jl) to run Jupyter notebooks on Travis, and make sure that the examples aren’t broken.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [November 26, 2017, 1:53am UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/6 "2017-11-26T01:53:46Z")

</div>

In Query I have an `examples` folder, and run them as part of my tests. I’m not super happy with that because it really slows down the build, but at least I know when something breaks.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [November 26, 2017, 2:02am UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/7 "2017-11-26T02:02:08Z")

</div>

Thanks for sharing this @cstjean, can be quite useful!

---

<div class="post-metadata">

**Author:** ![Evizero](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/evizero/32/10118_2.png) [@Evizero](https://discourse.julialang.org/u/Evizero)\
**Post date:** [November 26, 2017, 9:03am UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/8 "2017-11-26T09:03:49Z")

</div>

> [@Tamas\_Papp](#):
>
> Is there a way I can splice a file into documentation generated with Documenter.jl as Julia source code, ie between ```s?

I have somewhat of a hack at Augmentor.jl to convert plain `examples/*.jl` scripts into nice looking documentation pages as well as jupyter notebooks. Its more of a curiosity than a clean solution but it does its job quite nicely

[https://github.com/Evizero/Augmentor.jl/blob/master/docs/exampleweaver.jl](https://github.com/Evizero/Augmentor.jl/blob/master/docs/exampleweaver.jl)

example input script: [https://github.com/Evizero/Augmentor.jl/blob/master/examples/mnist\_tensorflow.jl](https://github.com/Evizero/Augmentor.jl/blob/master/examples/mnist_tensorflow.jl)  
output doc page: [https://evizero.github.io/Augmentor.jl/generated/mnist\_tensorflow/](https://evizero.github.io/Augmentor.jl/generated/mnist_tensorflow/)  
output notebook: [https://nbviewer.jupyter.org/github/Evizero/Augmentor.jl/blob/gh-pages/generated/mnist\_tensorflow.ipynb](https://nbviewer.jupyter.org/github/Evizero/Augmentor.jl/blob/gh-pages/generated/mnist_tensorflow.ipynb)

---

<div class="post-metadata">

**Author:** ![bramtayl](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bramtayl/32/3614_2.png) [@bramtayl](https://discourse.julialang.org/u/bramtayl)\
**Post date:** [November 27, 2017, 3:24am UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/9 "2017-11-27T03:24:17Z")

</div>

With a bit of rejiggering, you can test doctests as part of Pkg.test. You just have to move the code for building your documentation inside your testing file (set strict = TRUE and also some messing around with file paths). I do this for all of my packages, and try to have as many tests in the form as doctests as possible.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 27, 2017, 6:49am UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/10 "2017-11-27T06:49:07Z")

</div>

Thanks! Can you link an example of testing standalone examples? I looked at some of your packages and saw how you build the docs in `runtests.jl`, but found no example testing.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 27, 2017, 6:50am UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/11 "2017-11-27T06:50:28Z")

</div>

While I got many helpful answers for questions 2 and 3, no one said anything about 1, so I guess there is no standard location for examples. I will just use `examples/`.

---

<div class="post-metadata">

**Author:** ![bramtayl](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bramtayl/32/3614_2.png) [@bramtayl](https://discourse.julialang.org/u/bramtayl)\
**Post date:** [November 27, 2017, 8:44am UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/12 "2017-11-27T08:44:55Z")

</div>

So for example NumberedLines has no proper tests, only doctests. Nevertheless coverage is 96%. [https://bramtayl.github.io/NumberedLines.jl/stable/](https://bramtayl.github.io/NumberedLines.jl/stable/)

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [November 27, 2017, 9:14am UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/13 "2017-11-27T09:14:40Z")

</div>

Maybe there should be a standard? I remember a time when we didn’t have a real standard for the `src` and `test` folders, now there is. Maybe a similar standardization for the examples would be good?

---

<div class="post-metadata">

**Author:** ![Evizero](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/evizero/32/10118_2.png) [@Evizero](https://discourse.julialang.org/u/Evizero)\
**Post date:** [November 27, 2017, 9:21am UTC](https://discourse.julialang.org/t/best-practices-for-examples-and-ci/7304/14 "2017-11-27T09:21:35Z")

</div>

As far as I can tell, `examples/` seems like an unspoken standard