Wait. What you are looking for are Doctests, right? These are available in Documenter.jl:
https://juliadocs.github.io/Documenter.jl/stable/man/doctests/#Doctests-1
I think thereās a misunderstanding here - Iām not against having examples in the documentation for the user to run themselves; What Iām saying is that those examples for the user are a different thing to āautomatically running tests on a packageā, which is what CI is for. Those examples are used as an introduction to a user, while actual unit tests are for testing whether the package itself works as intended by the author and does what itās supposed to do.
In regards to running those examples automatically - Iām not quite sure what you mean here. Are you thinking about running those examples and showing the user the output?
Thank you, thatās a useful answer!
However, Documenter.jl says to use jldoctest while the Julia docs says to use julia-repl for examples. So I am still a bit confused about what to do.
I am just saying that in a first approach having as CI tests running the examples in the doc is better to have no tests at all. I have been maintaining huge packages in GAP3 with only such tests, and with a good enough documentation this is very useful. Most Julia docs I have looked at are in contrast very terse and contain few if any examples so they are not suitable.
Iām not familiar with GAP3, are you talking about something like this?
Ah, I understand now. Yes, that indeed is what Documenter can do. Having those running is better than no tests at all, but theyāre certainly not the only thing that should be run when testing a package (in the software development sense, not in the sense of trying a package out to see if itās a good fit). If youāre just āshopping aroundā for packages, a good introduction to that package is of course necessary - but again, thatās a different thing to automatically testing whether the package works as intended.
In any case (and Iām aware that this is a very subjective topic and that this is just an opinion), I donāt think just running examples from documentation is providing good enough test coverage for the actual code and functionality. Not every possible combination of arguments for a function can or should be presented in documentation, and often there are absurd weird combinations that trigger a bug - fixing those and writing a unit test to detect if that bug shows up again in the future is very much a necessity, but shouldnāt necessarily be documented in reference manuals. Especially invalid input and error states are things to be tested in unit tests, to ensure proper function of the code while not really suitable for documentation.
The bare minimum for a julia package is still just basic git version control. Neither CI nor documentation is technically required. But itās very much recommended to ensure much easier maintainability and usability of a package.
Yes
And of course additional tests may be needed than the documentation, but the documentation may contain
90% of the necessary tests which thus donāt have to be written twice.
Documenter.jl is an additional package so its clear that the julia documentation will not mention it. I have no experience in running the doctest but Documenter is usually on travis after the regular test. I am right now not sure if they influence the CI result (pass/fail).
Depends on your script/setup. You can run do it either way.