Literate.jl: how to skip code parsing/testing for quick generation and preview of the documentation?

Tooling
,
1

I am using Literate —> Documenter for the documentation of my package, but the inclusion of some tutorials makes the generation of the documentation taking a while.

Sometimes I would just like to try how the documentation (including the automatic links) is rendered… Is there some keyword I can pass to Literate.Markdown to tell it to skip the code evaluation/testing (or perhaps in a preprocess function ???) ? Then I could call my make.jl script with a keyword like “preview” or “quick” to speed up the generation of the documentation.

This is my current make.jl:

using Documenter, Literate, BetaML, Test

push!(LOAD_PATH,"../src/")


const _TUTORIAL_DIR = joinpath(@__DIR__, "src", "tutorials")
const _TUTORIAL_SUBDIR = [
    "Getting started",
    "Regression - bike sharing",
    "Classification - cars",
    "Clusterisation - Iris"
]

function link_example(content)
    edit_url = match(r"EditURL = \"(.+?)\"", content)[1]
    footer = match(r"^(---\n\n\*This page was generated using)"m, content)[1]
    content = replace(
        content, footer => "[View this file on Github]($(edit_url)).\n\n" * footer
    )
    return content
end

function _file_list(full_dir, relative_dir, extension)
    return map(
        file -> joinpath(relative_dir, file),
        filter(file -> endswith(file, extension), sort(readdir(full_dir))),
    )
end

"""
    _include_sandbox(filename)
Include the `filename` in a temporary module that acts as a sandbox. (Ensuring
no constants or functions leak into other files.)
"""
function _include_sandbox(filename)
    mod = @eval module $(gensym()) end
    return Base.include(mod, filename)
end

function literate_directory(dir)
    rm.(_file_list(dir, dir, ".md"))
    for filename in _file_list(dir, dir, ".jl")
        # `include` the file to test it before `#src` lines are removed. It is
        # in a testset to isolate local variables between files.
        @testset "$(filename)" begin
            _include_sandbox(filename)
        end
        Literate.markdown(
            filename,
            dir;
            documenter = true,
            postprocess = link_example,
        )
    end
    return nothing
end


literate_directory.(joinpath.(_TUTORIAL_DIR, _TUTORIAL_SUBDIR))


makedocs(sitename="BetaML.jl Documentation",
         authors = "Antonello Lobianco",
         pages = [
            "Index" => "index.md",
            "Perceptron" => "Perceptron.md",
            "Trees" => "Trees.md",
            "Nn"   => "Nn.md",
            "Clustering" => "Clustering.md",
            "Utils" => "Utils.md",
            "Tutorials" => map(
                subdir -> subdir => map(
                    file -> joinpath("tutorials", subdir, file),
                    filter(
                        file -> endswith(file, ".md"),
                        sort(readdir(joinpath(_TUTORIAL_DIR, subdir))),
                    ),
                ),
                _TUTORIAL_SUBDIR,
            ),
            "Examples" => "Examples.md"
         ],
         format = Documenter.HTML(prettyurls = false)
)
deploydocs(
    repo = "github.com/sylvaticus/BetaML.jl.git",
)
2

I did try with doctest = false in the (Documenter) makedocs function, but without success… it seems the code is parsed and executed in Literate, before reaching Documenter…

3

Unless you pass execute=true to Literate.markdown Literate does not execute any code. By default Literate results in Documenter @example blocks though, which are executed by Documenter. Documenter does not have a preview/draft option (might be a good feature request), but you can tell Literate to not generate @example blocks using the codefence keyword argument. For example, to output generic julia code blocks you can pass

codefence = "```julia" => "```"

to Literate.markdown.

As a sidenote, why do the code take so long? You could probably speed things up a lot by using Revise (unless the time is actually dominated by runtime rather than compile time).

1 Like
4

I added as you suggested the parameter codefence = "```julia" => "```" in the Literate.markdown call in the make script above, but still the code is computed…

image
image1178×379 55.4 KB

5

Can you link to the repo? I don’t see any code execution from Literate in that screenshot.

6

The repository is here: https://github.com/sylvaticus/BetaML.jl
And, in particular, the documentation is here and the make.jl (without yet the codefence committed) is here.

7

Thank you, I solved.

It was due to this:

"""
    _include_sandbox(filename)
Include the `filename` in a temporary module that acts as a sandbox. (Ensuring
no constants or functions leak into other files.)
"""
function _include_sandbox(filename)
    mod = @eval module $(gensym()) end
    return Base.include(mod, filename)
end
[...]
# `include` the file to test it before `#src` lines are removed. It is
# in a testset to isolate local variables between files.
@testset "$(filename)" begin
     _include_sandbox(filename)
end

Actually I understood now that the code was running twice! Once for the above testset, and once as an example block in Documenter.
By commenting the code above and using the codefence keyword in Literate.Markdown I managed to get the documentation buildwithout running the code, thanks.