Literate.jl to PDF using Documenter.jl simple question

simple question: how do I take the markdown output of Literate.jl and feed it to Documenter.jl to create a PDF (with Latex working)? The documentation of Literate.jl makes sense, but Documenter.jl is very complicated.

I can see that Literate.markdown("psm_test.jl") creates a psm_test.md file, but then what is the Documenter.jl command to turn this into a PDF (calling LaTeX on the formula?) I really did try reading the Literate.jl and Documenter.jl docs for over an hour… thanks in advance.

julia> Literate.markdown(“psm_test.jl”)
[ Info: generating markdown page from ~/perrin2013/SCaRF_Julia/literate/psm_test.jl
[ Info: writing result to ~/perrin2013/SCaRF_Julia/literate/psm_test.md
“/Users/perrin/perrin2013/SCaRF_Julia/literate/psm_test.md”

#======================================================================
ok, Julia has multi line comments pound equals ... equals pound...  cool

Literate.jl should turn this into markdown, so this should be **BOLD**

This next math equation is from the Literate.jl documentation 

```math
\int_\Omega \nabla v \cdot \nabla u\ \mathrm{d}\Omega = \int_\Omega v f\ \mathrm{d}\Omega
```

what is magic incantation of Documenter.jl to turn this into a PDF? 
===========================================================================#
using WAV

y,Fs,nbits,stuff = wavread("JolieHollandSacha.wav") # 16bit 44_100Hz sampling (ripped from CD) 

y_int_24 = floor.(Int32,y * 2^23)

wavwrite(y_int_24,"test_WAV_dot_jl__24bit.wav",Fs=44_100,nbits=24,compression=WAVE_FORMAT_PCM)

You use Literate as a sort of preprocessor to produce the .md file, which you then include in your docs exactly as you would include a .md file that you had written out manually. Then, if you want a PDF version, you can just follow these instructions to build via LaTeX.

I would suggest building up to a full PDF version. First, make sure you can generate the HTML, since that’s the most basic way to use Documenter, and will make sure you have the Literate+Documenter things set up properly before trying to involve LaTeX. I’m not sure if this is the simplest way, but I think Documenter requires some basic structure, so I followed the guide and put the following in Example/src/Example.jl:

module Example

export func

"""
    func(x)

Return double the number `x` plus `1`.
"""
func(x) = 2x + 1

end

I then put this in the Example/docs/src/index.md (because it’s mandatory for HTML output, though not for LaTeX):

# Example.jl Documentation

```@docs
func(x)
```

I then put your file in Example/src/psm_test.jl, as well as the following in Example/docs/make.jl:

# I haven't made `Example` an actual package that can be installed, so I just do this:
push!(LOAD_PATH,"../src/")

using Documenter, Example, Literate

# Use Literate to convert `psm_test.jl` to markdown for Documenter
input_file = "../src/psm_test.jl"
output_directory = "src/"
Literate.markdown(input_file, output_directory; documenter=true, execute=false)

# Now make the docs with Documenter
makedocs(
    sitename="My Documentation",
    remotes=nothing,  # Don't look for a git repo of the `Example` package; I haven't made one
    draft=true,  # Don't run the `@example` block; I don't have the necessary files, etc.
    pages=[
        "Introduction" => "index.md",
        "PSM Test" => "psm_test.md",
    ],
)

Note that I’ve had to add some hacky lines in there because I didn’t make Example a real project. If you have a real project, install it into the docs project, and you can remove the LOAD_PATH and remotes lines. Also, since you’re evidently able to run the example, you should be able to delete the draft line — though again, I’d only do that after getting it running in draft mode.

Then, from the Examples/docs directory, I run

julia --project make.jl

(You may have to install Documenter and/or Literate in the docs project manually; I actually have both in my base project, so this just works for me.) This generates some warnings because I haven’t made this into an actual package, but the build succeeds, and I can

open build/psm_test/index.html

which has the desired docs.

Next, just build the .tex file itself because you might not have all the LaTeX stuff installed, or Documenter might not know where to find it, but the .tex file should build successfully anyway. To do so, just add

    format=Documenter.LaTeX(platform="none"),

to the makedocs call and run again. This time, you should find Example/docs/build/MyDocumentation.tex containing the complete docs. (You can remove the index.md line now, if you want.)

Finally, if that looks sensible, you can have Documenter actually call LaTeX by removing platform="none" from the line above, so that it now looks like

    format=Documenter.LaTeX(),

And you should have Example/docs/build/MyDocumentation.pdf.

If any of these steps fails, please feel free to ask more specific questions.

I sincerely thank you for trying to help, but I am really looking for a simple solution for self-documentation… (see error below). As I do numerical experiments in Matlab or Julia, I have for 20 years been embedding LaTeX in the comments, so I can figure out what I was trying to do many years later. I just pulled up a Matlab filter design I did from 15 years ago, and I ran it through my homegrown “literate matlab” C(!) code that strips LaTeX out of Matlab comments and runs them through pdflatex. I find this very helpful, and in the spirit of Knuth’s cweb and cweave programs, which he wrote for himself. Since I am no longer in graduate school I do not have the time (nor, frankly, the patience) to wade through Documenter.jl and figure out how to get it to pretty print Julia flavored Mardown with embedded LaTeX for a single file. (I Just modified my 20yo C code to strip out Julia comments out and feed them to pdflatex in less time than I have spent reading the Documenter.jl documents). But Markdown is slicker than LaTeX for text comments, and I just wrap Julia code in `\Verbatim{}’ so I don’t get the pretty colors or automatic formatting or hyperlinks… But it is simple, and it worked 15 years later…

IMHO, one of the keys to good numerical documentation is to get in the habit of documenting code as one writes it, even one-offs during numerical experiments. So in the spirit of Knuth’s Literate Programming, I think a function such as

Literate.renderJuliaMarkdown("psm_test.md", ) ; # creates psm_test.pdf and psm_test.html by default, LaTeX assumed

would be true to Knuth’s original intent for Literate programming. And the Julia user base is the perfect target for Literate Programming!

If I ever decide to release a Julia package for public consumption, of course I would use the excellent and powerful Documenter.jl package(I understand the power there, and I appreciate the Julia packages that use Documenter.jl for good documentation with properly typeset LaTeX equations.

Since I am looking for longevity, I can’t use notebooks, as I was bitten by the notebook bug 15 years ago, and of course any notebooks I have from back in the day are completely outdated. LaTeX and C and Matlab will be around in 10 years, I’m hoping Julia is too…

Thanks again for trying to help.

ERROR: LoadError: UndefVarError: `outputdirectory` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
Stacktrace:
 [1] top-level scope
   @ ~/perrin2013/SCaRF_Julia/literate/discourse_help/Example/docs/make.jl:9
in expression starting at /home/perrin/perrin2013/SCaRF_Julia/literate/discourse_help/Example/docs/make.jl:9
[1]+  Done                    emacs "$1" -foreground green -background black
perrin@ubuntu20:~/perrin2013/SCaRF_Julia/literate/discourse_help/Example/docs$ 

Sorry, that outputdirectory should have been output_directory — which was defined on the line immediately before it.

I understand that not everyone’s workflow is the same, but I would bet that if you just put in a tiny bit more effort to get everything working nicely, you would be satisfied with the results.

Most of us here are fluent in LaTeX, and have also been using it for decades. But the full machinery does not fit well with modern applications, like documenting code in ways that interact well with other tools. Most of us feel strongly that Documenter does an excellent job of filling the gap, but it’s extremely powerful and therefore has a lot of options you can tweak. For example, you can use Julia from within the documentation, and automatically test to ensure that your documentation actually agrees with the code.

Literate is a relatively lightweight package with limited scope — basically it’s doing little more than your strip-comments-and-verbatimize-code step. It doesn’t do any of the processing that Documenter does, which is more akin to latex itself.

That said, maybe @fredrikekre would have some thoughts about the possibility of a wrapper for quick-and-easy tests. I guess it is true that most of this machinery is designed to document packages, rather than scripts; maybe there’s something that could be improved there.

Just to be clear, I would hope this would call Documenter.jl under the hood (and whatever version of LaTeX its heart desires), I agree that Markdown and the modern tooling far better than LaTeX for almost everything except the mathematics, at which LaTeX is the math community standard and probably will be for a while

But back to Documenter.jl… now I get this error(?)

perrin@ubuntu20:~/perrin2013/SCaRF_Julia/literate/discourse_help/Example/docs$ julia --project make.jl
[ Info: generating markdown page from `~/perrin2013/SCaRF_Julia/literate/discourse_help/Example/src/psm_test.jl`
[ Info: writing result to `~/perrin2013/SCaRF_Julia/literate/discourse_help/Example/docs/src/psm_test.md`
[ Info: SetupBuildDirectory: setting up build directory.
[ Info: Doctest: running doctests.
[ Info: ExpandTemplates: expanding markdown templates.
[ Info: CrossReferences: building cross-references.
[ Info: CheckDocument: running document checks.
[ Info: Populate: populating indices.
[ Info: RenderDocument: rendering document.
[ Info: HTMLWriter: rendering HTML pages.
┌ Warning: Unable to determine the repository root URL for the navbar link.
│ This can happen when a string is passed to the `repo` keyword of `makedocs`.
│ 
│ To remove this warning, either pass a Remotes.Remote object to `repo` to completely
│ specify the remote repository, or explicitly set the remote URL by setting `repolink`
│ via `makedocs(format = HTML(repolink = "..."), ...)`.
└ @ Documenter.HTMLWriter ~/.julia/packages/Documenter/iRt2s/src/html/HTMLWriter.jl:763
┌ Warning: Cannot extract version for inventory from /home/perrin/perrin2013/SCaRF_Julia/literate/discourse_help/Example/Project.toml: no such file
└ @ Documenter.HTMLWriter ~/.julia/packages/Documenter/iRt2s/src/html/write_inventory.jl:89
┌ Warning: Please set `inventory_version` in the `HTML()` options passed to `makedocs`.
└ @ Documenter.HTMLWriter ~/.julia/packages/Documenter/iRt2s/src/html/write_inventory.jl:98
[1]+  Done                    emacs "$1" -foreground green -background black

and so closer but still no cigar. I will now stop shamelessly self-bumping this topic and hope that I’ve made my point. Thanks again for your help, I feel I’m a lot closer now, but for single scripts it still seems like too much of a lift.

Tantalizingly close…


but alas when clicking on PSM Test

its not rendered in HTML, let along LaTeX…

using Documenter, Literate

Literate.markdown("./litdocpdf.jl", "./md"; flavor = Literate.CommonMarkFlavor()) 
# CommonMarkFlavor because all we need is syntax highlighting
# making them @example blocks (via DocumenterFlavor) just needlessly complicates things

makedocs(; remotes = nothing, format = Documenter.LaTeX(), sitename = "PSM Test", pages = ["litdocpdf.md"], pagesonly = true, source="md")

This created this PDF from your code in the first post.

You can also add an authors kwarg to the makedocs call to add authors to the PDF title page. For more customization (eg. to remove the “table of contents”), I believe you’d have to add an assets folder under md and add a preamble.tex to it like this with your customizations.

(I also needed the Python package Pygments installed - I did it via pipx - and since this was a new-ish system without a full LaTeX installation, needed to add the newunicodechar, polyglossia, minted, tabulary, adjustbox, and tcolorbox packages for it to generate the PDF.)

2 Likes

Sure, but LaTeX is allowed within markdown. For Julia’s flavor, you can use either inline and display form. That’s no problem.

Those aren’t errors, just warnings. As I said, Documenter does assume that you’re documenting a package, so it looks for all the things that should be present in a package so that it can do nice things like include the version and link to the repo. If it can’t find those, it warns you so that you don’t get incomplete documentation.

It is actually in HTML, but you’re not serving the HTML; you’re just opening files with your browser. And your browser is doing what browsers do with plain files. If you click on the index.html file, you should see it rendered.

My Bad. I did not notice the index.html file in Example/docs/build/psm_test (it was not linked from the “main” index.html for some reason, and I was too quick to be frustrated, I apologize).

when I click on it, I GET EXACTLY WHAT I WANT! THANKS!

For a single Julia script file this is a little cumbersome, I still think it would be awesome if there were a function such as

Literate.renderSingleFileJuliaMarkdown("psm_test.md") 

that created this directly by calling Documenter.jl under the hood. (just the white bit).

Now I’m curious how Documenter.jl renders LaTeX so nicely in HTML, I’m impressed. It’s not the easiest to read HTML, but it appears as if javascript is actually rendering LaTeX “on the fly” (I assumed it was embedded GIFS, as was done 15 years ago… However, it might be slightly “fragile” (i.e in 15 years if I click on that index.html file, what are the odds that it will render?). So I still think a PDF option would be ideal for posterity purposes. (Yes, I know there must be a way to get Documenter.jl to output PDF, I just mean the simple hypothetical function I am proposing. Perhaps my icon photo is misleading, I’m 54 years old, I graduated from Courant in 2001, and I have been using Matlab since 1988 and Linux since 1993… I have noticed that when I browse through my code archives, there a lot of files that I can’t open anymore (SigmaPlot files are the one that hurt the most, I spent a lot of time getting those things to look good). Anyone here remember the commercial plotting program SigmaPlot? In the 1990s it was the only way to produced decent scientific plots, GNUplot did not have great output). Of course MatPlotLib is awesome these days. Makie.jl is a bit like documenter.jl, obviously super powerful but it changes so quickly that my Makie plots from three years ago already throw multiple errors…

Thanks again for your help.

Thanks, I’m really not trying to be difficult, but when I view your PDF I get this:

but instead what I want is (see below):

The web has long since moved on from GIFs. Nowadays, it’s mostly JavaScript in the form of MathJax or KaTeX — both of which are supported by Documenter. The former is managed by the American Mathematical Society, and has been going strong for 16 years already. There are so many users for these packages that I can’t imagine either will be unavailable 15 years hence.

That’s fair. They’re just used for different things.

Did you scroll to page 3 of that PDF? It shows almost exactly what you want.

As @moble mentions (and the (1 of 3) in the top left of your image indicates), the main content is on page 3, and what’s shown in your image is the title page. If you don’t want the title and ToC pages (understandably), you can add an assets/preamble.tex file to override the default preamble (which adds the title page and Table of Contents) and instead make it use your own LaTeX preamble.

My Bad again. Yes, perfect. Sincere thanks.

Thanks! I found the Documenter.jl preamble.tex, and commented out the TOC, put it in md/assets and now I get a single page PDF! Perfect.

when I try to use your script to generate a HTML file (which I know is possible…),

makedocs(; remotes = nothing, format = Documenter.HTMLWriter(), sitename = "PSM Test", pages = ["psm_test.md"], pagesonly = true, source="md")

I get a complaint about Module. Is there an easy way to create a single-page HTML file equivalent to the single page PDF?

ERROR: LoadError: MethodError: objects of type Module are not callable
The object of type `Module` exists, but no method is defined for this combination of argument types when trying to treat it as a callable object.
Stacktrace:
 [1] top-level scope
   @ ~/perrin2013/SCaRF_Julia/literate/discourse_help/digital_carver/readme.jl:11
in expression starting at /home/perrin/perrin2013/SCaRF_Julia/literate/discourse_help/digital_carver/readme.jl:11
perrin@ubuntu20:~/perrin2013/SCaRF_Julia/literate/discourse_help/digital_carver$ 

Thanks again

format should be HTML not HTMLWriter.

makedocs(; remotes = nothing,
                   format = Documenter.HTML(; prettyurls = false, edit_link = nothing, repolink = nothing, inventory_version = ""),
                   sitename = "PSM Test", pages = ["litdocpdf.md"], pagesonly = true, source="md")

These options minimize unnecessary warnings (there’s still one warning, that can be ignored), and produce a single HTML file output.

(There’s still a navigation sidebar on the left in this HTML page, even though it isn’t necessary given that it’s a single page - that part doesn’t seem configurable.)

Thank you!!! Literate.jl and Documenter.jl are certainly amazing tools for mathematical documentation, and they are aesthetically beautiful! They complement the elegance of Julia
notation nicely (as apposed to Python/Numpy abominations…).