Hi all,
for Jupyter notebooks there is a tool called “nbval”, that can re-run the notebook and check that the new output is the one that is saved in the notebook itself.
Thanks to this, notebooks become useful as an integration/regression tests.
I think that this can be achieved in Julia with Documenter.jl using jldoctest blocks, but the interface to write the document is not as convenient as a notebook interface that just produce the output.
On the other hand, AFAIK Pluto does not save outputs, so such a check is impossible. I would like my last sentence to be refuted:
Is there a way to save the output in Pluto notebooks?
If yes, is there a way to check that re-running the notebook produces the same result?
If not, are there other tools that allow this, except jupyter + julia kernel?
would exporting to PDF/HTML and then taking a diff be practical? Additionally: is there a way to load a Pluto notebook, run its content and save it as PDF/HTML in a “headless” way, so that this could be done in an automatic fashion?
According to this thread these seem to be the alternatives, but I might be missing something and in the last years things might have evolved - that is why I am asking.
which is a very nice way of testing outputs.
This does not exactly answer your question,
because tests have to be written manually,
but the interface is awesome.
I used fine tools like nbdiff, and now like the PlutoTest workflow better actually.
The notebook can also be included in a runtests.jl, e.g.:
@testset objectives begin
include("notebooks/objectives.jl")
end
So all tests can be checked in one go,
and if one test is failing,
the issue can be investigated in the corresponding notebook.
Is there a way to save the output in Pluto notebooks?
It depends what you mean by “output”. You can have code in the notebook which saves some data, or you can export to pdf or html.
If yes, is there a way to check that re-running the notebook produces the same result?
IMHO the best way is to have some inbuilt tests.
If not, are there other tools that allow this, except jupyter + julia kernel?
would exporting to PDF/HTML and then taking a diff be practical?
I am not sure about this, because this output very much depends on the tools used to generate it. E.g. I guess HTML output contains a lot of HTML code dependent on the particular Pluto version.
Additionally: is there a way to load a Pluto notebook, run its content and save it as PDF/HTML in a “headless” way, so that this could be
done in an automatic fashion?
Yes, there are two packages allowing for this:
PlutoStaticHTML.jl generates html pages from Pluto notebooks which feel not “Pluto-like”
PlutoSliderServer.jl allows for headless export of Pluto notebooks very much like the export generated from Pluto itself
(I did not try PlutoTest.jl)
As for testing, I personally prefer to have tests in the notebook. As Pluto notebooks are Julia scripts you can run them during CI just as scripts.
I built some infrastructure around this, see ExampleJuggler.jl. There you can also find how to use the different ways to generate HTML from pluto.
Thank you for the suggestion. What I am trying to understand is if there is a way to use pluto notebooks as integration tests for a package which are also part of the documentation.
The idea is to have separate unit tests in a test suite, but have a notebook that shows how a package works. Having the tests in the documentation might make it a little harder to read.
The one I have in mind might not be the best use case for Pluto, since in this specific case reactivity is not needed at all and it could even be detrimental (In this case I don’t want the output to change automatically, I want a system that warns me when the output is different from the documented one so that I can go and check why).
This is possible with Documenter and jldoctest. The Documenter file can be even written before the code exists. But if the code already exists, it can be useful to write a tutorial in a notebook, and use it as a regression test. This can be done with jupyter and nbval, but Pluto is cooler and (I would argue) more popular than jupyter in the Julia community, so I wanted to understand if perhaps one could just do the same without jupyter.
In any case, thank you both for pointing me at PlutoTest.jl and ExampleJuggler.jl,
and also for this remark:
I am not sure about this, because this output very much depends on the tools used to generate it. E.g. I guess HTML output contains a lot of HTML code dependent on the particular Pluto version.