Literate.jl announcement

[Wenjie]

Awesome stuff! However, I wonder if it can be used in scenarios other than generating package documentation and standalone notebooks. In particular, I would like to know whether it can be used in the following situations:

1. If an existent package doesn’t use Literate.jl to generate its documentation, and that I want to contribute to its documentation, could I still use Literate.jl?

2. If I want to report an issue in Github, can I use the generated output of Literate.jl?

3. Can it cooperate with Jekyll, which transforms texts to a website? With Jekyll, I only need to write a makedown file with some header in it, and the file will produce a corresponding html file automatically in the final website.

For the first 2 points, I am aware that even though we can, it might not be a good practice. If you find it troublesome to offer these options, just let us forget these ideas.

[Wenjie]

Just tried your packages. Please allow me to give some feedback.

I feel that your package could have had a wider usage if not having been limited within documentation and notebooks. Currently, if not sent to documenter.jl or converted to a notebook, the output document can’t have its code executed. What if I want only to send a standalone html file to someone? If the Markdown output can already get its code executed, all problems will be solved. With the execution results embedded in Markdown, we can convert it to HTML, rST, LaTeX, and so on.

To achieve this, I would suggest starting with:

1. add customized fence for math code as well;
2. extract the execution code from documenter.jl.

[Wenjie]

My point is to produce a self-complete markdown (with code already executed) and be able to use Pandoc afterwards.

[fredrikekre]

@Wenjie

1. If an existent package doesn’t use Literate.jl to generate its documentation, and that I want to contribute to its documentation, could I still use Literate.jl?

You mean to use Literate.jl to generate a markdown which you then contribute to the docs of the package? I mean, I guess you can, but that’s not really the point of the package, and would probably be more hassle then to just use whatever doc setup the package already have.

@Wenjie

If I want to report an issue in Github, can I use the generated output of Literate.jl?

I don’t really understand what you mean by this.

@Wenjie

Can it cooperate with Jekyll, which transforms texts to a website? With Jekyll, I only need to write a makedown file with some header in it, and the file will produce a corresponding html file automatically in the final website.

Yes, I don’t see a reason why not. That is essentially what Documenter does – takes the markdown file and generates html.

@Wenjie

I feel that your package could have had a wider usage if not having been limited within documentation and notebooks.

I don’t think it is limited to this, but yes, that was the goal of the package.

@Wenjie

1. add customized fence for math code as well;

Why do you need this? Literate.jl does not impose a specific math fence so you can use whatever you want.

@Wenjie

2. extract the execution code from documenter.jl.

Why don’t you wanna use Documenter.jl though? You can use its markdown generator, which will execute the code.
Source file:

# Hello

```@example
rand(2,2)
```

Resulting file:

<a id='Hello-1'></a>

# Hello


```julia
rand(2,2)
```

```
2×2 Array{Float64,2}:
 0.984576  0.51578
 0.220773  0.132578
```

[baggepinnen]

Is there a suggested strategy for capturing plots in the code, other than manually saving and including them in the markdown?

Edit: Also, is there a way to have code present that is run when the document is compiled but not included in the markdown output?

Edit2: It seems #src can be put in the end of a line to accomplish this!

[fredrikekre]

@baggepinnen

Is there a suggested strategy for capturing plots in the code, other than manually saving and including them in the markdown?

Documenter captures the plots, and includes it in the output, but only for svg format ATM (https://github.com/JuliaDocs/Documenter.jl/blob/6abfaeee9f488f0487a022551634807daf4c2515/src/Expanders.jl#L500-L503). The notebook generator does something similar, but should work for any format.

@baggepinnen

Edit: Also, is there a way to have code present that is run when the document is compiled but not included in the markdown output?

Edit2: It seems #src can be put in the end of a line to accomplish this!

Yep

[baggepinnen]

Thanks for your reply! I was considering a use case where I generate an md file as the final output, e.g. for uploading to github.
I’ll see if I can come up with a simple function that saves the current Plots figure and returns the file path to be included as a markdown link, possibly through the post processing functionality

[baggepinnen]

It took me a while to figure out that Literate doesn’t run any code, that’s up to Documenter.
I managed to get what I want with the use of Weave and the following helper function, which is called just like weave would be called

function superweave(source, args...; kwargs...)
    tmpname = tempname()
    Literate.markdown(source, tmpname, documenter=false)
    sourcename = replace(source, ".jl", ".md")
    sourcename = match(r"(/\w+.md)", sourcename)[1]
    sourcename = tmpname*sourcename
    jmdsource = replace(sourcename,".md", ".jmd")
    run(`cp $(sourcename) $(jmdsource)`)
    weave(jmdsource, args...; kwargs...)
end

julia> superweave("/tmp/test.jl", doctype="md2pdf")

[Wenjie]

Thanks for the reply! I see your main purpose to write this package is for education, whereas @baggepinnen and I are both thinking about production.

By suggesting using Documenter.jl, did you mean the makedocs function? I tried it. It worked, but it’s troublesome and error-prone.

I will try @baggepinnen’s example.

[fredrikekre]

@baggepinnen

I managed to get what I want with the use of Weave and the following helper function, which is called just like weave would be called

Why not use weave directly instead?

@Wenjie

By suggesting using Documenter.jl, did you mean the makedocs function?

Yes.

@Wenjie

I see your main purpose to write this package is for education , whereas at-baggepinnen and I are both thinking about production .

Well, I don’t know about that - Literate just transforms the code from one format to another, and you can use whatever markdown → some other format converter you want. I don’t see why this could not be used for production?

I kinda see Literate as a Documenter extension, so for now it relies on Documenter for code execution. But note that the code-executing machinery is already there (and used for the notebooks) so it should be trivial to implement something that executes a markdown file too. Feel free to make a PR!

[baggepinnen]

@fredrikekre

Why not use weave directly instead?

Because the .jmd files are not runnable, atom support for running cells feels a bit awkward and include() does not work either. The Literate + Weave workflow seems to involve much less maintenance for my use cases.

[Wenjie]

@baggepinnen

Because the .jmd files are not runnable

Just learned Weave, but it seems to me that it also supports the script input along with the .jmd format. See this paragraph in their website:

You can write your documentation and code in input document using Noweb, Markdown or script syntax and use weave function to execute to document to capture results and figures.

[Wenjie]

I feel the main advantage of your package over Weave is the customization it provides.

Weave provides end-to-end (i.e., input => output) solutions to existent formats. Whenever one wants to work with an uncommon format, Weave becomes useless.

Your pre-processing provides the conversion between input formats, and your post-processing provides the conversion between intermediate formats, which is Weave lack of.

I think Weave can use some of your ideas.

[fredrikekre]

@Wenjie

I feel the main advantage of your package over Weave is the customization it provides.

Weave provides end-to-end (i.e., input => output) solutions to existent formats. Whenever one wants to work with an uncommon format, Weave becomes useless.

Your pre-processing provides the conversion between input formats, and your post-processing provides the conversion between intermediate formats, which is Weave lack of.

Ok, thanks for the feedback btw.
So the main thing you think is lacking is the ability to execute code and include the result in the markdown file (i.e. without setting up Documenter)? Am I understanding you correctly?

@Wenjie

I think Weave can use some of your ideas.

I tried ([RFC] Custom pre- and post processing in convert_doc by fredrikekre · Pull Request #126 · JunoLab/Weave.jl · GitHub) before starting writing Literate

[Wenjie]

@fredrikekre

So the main thing you think is lacking is the ability to execute code and include the result in the markdown file (i.e. without setting up Documenter)? Am I understanding you correctly?

After learning your package more, I won’t call it a “lack”; it perfectly solves your problem. Concerning my usage, neither your package nor Weave perfectly meet my need, but after some tweak, both can work.

The two packages start from different mindsets (Weave uses the model of knitr), so, at this moment, I can’t tell which is better than the other. A safe practice would be maintaining the compatibility with one of the Weave formats, either by using the same format or by offering a converter.

[Wenjie]

Some updates:

1. The example of Weave doesn’t work for me on JuliaBox: it never finishes running.
2. I wrote a blog post to promote your package.
3. For my use case, the ability to execute code and customizable fences for code, output, and math in the final markdown file would save my time (I use Liquid tag {% %} instead of ``` ``` for code).
4. I didn’t execute MYVARIABLE = MYVALUE successfully. Maybe I missed something when using Literate.markdown.
5. It would be better if you can include a plot in your example.

[baggepinnen]

@fredrikekre

So the main thing you think is lacking is the ability to execute code and include the result in the markdown file (i.e. without setting up Documenter)? Am I understanding you correctly?

This would cover the vast majority of use cases for me.

Weave has the ability to create a pdf via tex(minted) etc. which is nice, but the atom markdown->pdf converter could to this as well, perhaps with less control over the output.

[fredrikekre]

@Wenjie

2. I wrote a blog post to promote your package.

Cool! Would you mind linking to the actual example from the docs (https://fredrikekre.github.io/Literate.jl/stable/generated/example.html) instead of copy-pasting it into the post? The example in the documentation have the links correct etc. Or even better, come up with your own example with things that are interesting to you!

@Wenjie

3. […] and customizable fences for code, output, and math in the final markdown file would save my time (I use Liquid tag {% %} instead of for code).

I am confused by this, you can already use custom code fences and use whatever math syntax you want?

@Wenjie

4. I didn’t execute MYVARIABLE = MYVALUE successfully. Maybe I missed something when using Literate.markdown .

Those are just placeholders to show off the pre/post processinc functionalities. I replace those here: https://github.com/fredrikekre/Literate.jl/blob/5143749c476b2b11663750e15e6c67565fdb2154/docs/make.jl#L8-L16

@Wenjie

5. It would be better if you can include a plot in your example.

Yea, that is planned. I just need a plotting library that works without hassle on Travis

@Wenjie

3. For my use case, the ability to execute code […]

@baggepinnen

This would cover the vast majority of use cases for me.

See execute markdown files · Issue #9 · fredrikekre/Literate.jl · GitHub

[piever]

@fredrikekre

Yea, that is planned. I just need a plotting library that works without hassle on Travis

Plots with the GR backend works on Travis without much setup (it does throw a fair amount of warnings during the Travis build but they seem harmless) and is able to save plots as pngs. Here there’s an example (and here is the corresponding markdown).

@fredrikekre

So the main thing you think is lacking is the ability to execute code and include the result in the markdown file (i.e. without setting up Documenter)?

Of course I can’t speak for other people but I personally think that it would be a very neat feature to be able to do that. There are some use cases of generating a markdown from a Julia scripts that do not belong to the documentation of a package:

• Hugo/Jekyll to create a blog post
• Remark to create a slideshow (see this discussion).

I’m not sure I understand Literate well enough to contribute on the “execute code in the markdown” part, but once that exists I’ll be happy to contribute the integration with Remark to create slideshows (should actually be pretty straightforward).

EDIT: added here a tiny Remark.jl package to create the presentation from the markdown.

[Wenjie]

@fredrikekre

Cool! Would you mind linking to the actual example from the docs (https://fredrikekre.github.io/Literate.jl/stable/generated/example.html) instead of copy-pasting it into the post? The example in the documentation have the links correct etc. Or even better, come up with your own example with things that are interesting to you!

The actual example from the docs was already linked and given credit, and the erroneous link was also fixed. You might not notice it because I fixed it right after I posted here.

That said, that blog post is just a mockup to show my use case of Literate: writing blogs (maybe also slides) without excluding the possibilities of converting it to notebooks. However, the current state of the package prevents me from doing so for the reason I will mention in the next paragraph. So, for now, I will temporarily put that blog post offline.

Currently, it is not beneficial for me to use your package given the tedious workflow. To write a blog post, I have to go through the following stages:

1. Put #'s before all markdown lines in the root file A;
2. Literate.markdown generates file B with the following fences:

    ```@example
    1 + 1
    ```

    ```math
    \int
    ```

3. Documenter.makedocs generates file C with the following fences:

    ```julia
    1 + 1
    ```

    ```
    2
    ```

    $$
    \int
    $$

4. I manually change the fences line by line since the string replacement can’t work here:

    {% highlight julia %}
    1 + 1
    {% endhighlight %}

    {% highlight markdown %}
    2
    {% endhighlight %}

    \[
    \int
    \]

The Step 4 is painful for a long document. Given this workflow may be repeated several times, it is eventually unworthy to do literate programming.

The best way to solve this problem is to let Documenter (or whatever) generate the desired fences at the first place.

P.S. The use case (mentioned by @piever) of creating slideshow is also great; I second this.