Typst and other alternatives to LaTeX

No, that’s right. Pandoc includes all the basic stuff. It can create tables, references, footnotes, headings, lists, etc., in any of its output targets from the same Markdown source. Hooks are for doing something custom; you write a program telling Pandoc how to translate some Markdown syntax into any output format that you need. Here I explain an elaborate example, a set of hooks I created to help generate a book about gnuplot; Pandoc + my hooks create the example illustrations from the code samples in the book and keep everything straight:

https://lee-phillips.org/panflute-gnuplot/

I’d also like to point out, because scripting was mentioned a few times, that the recommended version of LaTeX, LuaLaTeX, has Lua built in as a scripting language that has access to the TeX internals: LuaTeX comes of age [LWN.net]

6 Likes

Fair, but I guess my point is at the end of the day your pandoc hooks, even if they’re written in python, and your lovely lua code, still need to emit latex. For me at least, anything that’s more complicated than what markdown does natively is nigh inscrutable in latex, so this solution wouldn’t get me very far.

LuaTeX may be used for creating documents from Lua directly. Or XML, etc. Too bad that they chose to base the whole thing on Lua.

But isn’t Lua a very capable and complete scripting language?

1 Like

Lyx is amazing. I have been pushed it in my group for many years.

I think that blog post is a great basis for discussion. Here’s a quick attempt to recreate these Pandoc filters/templates in Typst, for comparison:

Converting bold face to italic

Panflute:

# !/usr/bin/python3
import panflute as pf

def action(elem, doc):
    if isinstance(elem, pf.Strong):
        return pf.Emph(*elem.content)

if __name__ == '__main__':
    pf.toJSONFilter(action)

Save above as myfilter.py and compile doc with pandoc --filter myfilter.py ...

Typst:

#show strong: it => emph(it.body)

Add this line at the top of the document, compile as usual.

Defining blocks for comments and format-dependent output

Panflute filter to suppress code blocks with n language, and to add custom formatting to pubnote blocks but only for HTML output:

import panflute as pf

def action(elem, doc):
    if (isinstance(elem, pf.Code) or
        isinstance(elem, pf.CodeBlock)):
        if ('n' in elem.classes):
            return []
    if isinstance(elem, pf.CodeBlock):
        if 'pubnote' in elem.classes:
            if doc.format == 'html':
                return pf.convert_text(
                  '<div class = "pubnote">{}</div>'
                  .format(elem.text))
            else:
                return []

if __name__ == '__main__':
    pf.toJSONFilter(action)

Typst has built-in syntax for comments. On the other hand it doesn’t expose the output format (see here). Anyway here’s a way to suppress n blocks and to format pubnote blocks differently when --input format=svg is passed on the command-line:

#show raw.where(lang: "n"): none
#show raw.where(lang: "pubnote"): it => {
  set text(red) if sys.inputs.at("format", default: "pdf") == "svg"
  it
}

Custom quote block with parameters

Panflute:

import panflute as pf

def epigraph(options, data, element, doc):
    return pf.convert_text(
     '<div class = "epigraph">{}
      <span class = "who">{}</span></div>'
      .format(options.get('quote'), options.get('who')))


if __name__ == '__main__':
    pf.toJSONFilter(pf.yaml_filter,
       tags = {'epigraph': epigraph})

used in:

~~~ epigraph
quote: Simplicity is the ultimate sophistication.
who: Leonardo da Vinci
---
~~~ 

In Typst you’d simply define a function to format the epigraph:

#let epigraph(quote: none, who: none) = block[
  #quote

  #h(1fr)---#who
]

Used like this:

#epigraph(
  quote: [Simplicity is the ultimate sophistication.],
  who: [Leonardo da Vinci],
)

though I’d probably define the method without keyword arguments: #let epigraph(who, quote) = ... and call it like this:

#epigraph[Leonardo da Vinci][
  Simplicity is the ultimate sophistication.
]

Template using metadata from YAML file

Panflute:

<!DOCTYPE HTML>
<html dir="ltr" lang="en-US">
<head><meta content="text/html;charset=utf-8"
      http-equiv="Content-Type" />

    <title>$title$</title>

</head>
<body>

 <h1>$title$</h1>
 
 $if(related)$
  <div style = 'font-size: 0.8rem; color: green;'>
   <h2>Related articles:</h2>
   $for(related)$   
    <p>$related.title$: $related.url$</p>
   $endfor$
  </div>
 $endif$

 $body$

</body>
</html>

used in:

---
title: The History of Semicolons
author: Prof. Lexi Graphical
related:
- title: On Neglected Punctuation
  url:   "http://example.com/neglect/"
- title: "Semicolons: Can We Have Too Many?"
  url:   "http://example.com/yes.html"  
---

Semicolons are one of our most important,
yet most misunderstood punctuation marks. 

Typst doesn’t support HTML output yet, but here’s a template that works for PDF:

#let template(md, body) = [
  = #md.title

  #if "related" in md [
    #set text(0.8em, green)
    == Related articles:
    #for (title, url) in md.related [
      #title: #url
      #parbreak()
    ]
  ]

  #body
]

used in:

#show: template.with(yaml("file.yaml"))

Semicolons are one of our most important,
yet most misunderstood punctuation marks. 

(Here I load the YAML from a file but you can also pass the content inline to yaml.decode.)

Formatting gnuplot source code and results

In this example, gnuplot code blocks are processed by Panflute to put some lines in bold face and to execute the code and show the result side by side with the source.

Panflute:

# !/usr/bin/python3
import panflute as pf
import subprocess
c = subprocess.run
import re
import zlib
import os

def action(elem, doc):
    if isinstance(elem, pf.CodeBlock):
        if 'gnc' in elem.classes:
            # pff will hold the checksum
            pff = str(zlib.adler32(bytes
             (elem.text.replace('@', ''), 'utf-8')))
            #The name used for the
            #gnuplot output image:
            pfn = pff + '.png' 
            #Check if we've done this one: 
            if pfn not in os.listdir():
                dscript = elem.text.replace('@', '')
                script = '''set term pngcairo\n
                 set out "{}"\n{}'''.format(pfn, dscript)
                with open(pff + '.gn', 'w') as scriptfile:
                    scriptfile.write(dscript)
                #Execute gnuplot on the script:
                c('''echo '{}' | 
                 gnuplot'''.format(script), shell = True)
            if doc.format == 'latex':
                #Lot's of escaping needed:
                mt = elem.text.replace('{', '\\{')
                mt = mt.replace('}', '\\}')
                #LaTeX boldface:
                mt = re.sub('@(.*?)@', r'\\textbf{\1}', mt)
                mt = mt.replace('\\\n', '\\textbackslash\n')
                #for newlines in gnuplot labels, etc.
                mt = mt.replace('\\\\n', '\\textbackslash{n}') 
                return pf.RawBlock(
                '\n\hypertarget{'+pff+'}{}\\begin{Verbatim}\n'+mt+
                '\n\end{Verbatim}\n\\textattachfile[mimetype=text/'
                +'plain]{' + pff + '.gn}{' + '\\framebox'+'
                {Open script}}\n\plt{' + pfn + '}',
                format = 'latex')
            else:
                return [elem, pf.RawBlock(
                    '<br /><img alt = "" src = "' + pfn + '" />',
                    format = 'html')]

if __name__ == '__main__':
    pf.toJSONFilter(action)

used in:

\hypertarget{setting-ranges}{\subsection{Setting
Ranges}\label{setting-ranges}}

That was simple. Notice how gnuplot decided to plot
our function from -10 to +10. That's the default,
which we got because we didn't ask for any
particular range. Gnuplot also set the y-axis
limits (the range of the vertical axis) to
encompass the range of the function over that
default x-axis domain. Let's take control of the
limits on the horizontal axis (the new command is
highlighted). Gnuplot happens to know what π is
(but doesn't know any other transcendental
numbers).


\hypertarget{3260812063}{}\begin{Verbatim}
\textbf{set xrange [-pi : pi]}
plot sin(x)
\end{Verbatim}

\plt{3260812063.png}

Typst code cannot run external programs, but typst query could be used to extract the gnuplot code blocks for execution. typst compile could then be called use the files generated by gnuplot. But I find it simpler and cleaner to put all code blocks in a Jupyter notebook with the gnuplot kernel, so I’ll show a sketch of a solution based on this idea:

#import "@preview/based:0.1.0": base64
#let parse-line(line) = {
  let parts = line.split(" ## ")
  let code = parts.first().trim(at: end)
  if parts.len() == 1 {
    (code, ())
  } else {
    (code, parts.at(1).split(",").map(str.trim))
  }
}
#let parse-images(cell, ..args) = {
  cell.at("outputs", default: ())
    .filter(x => "image/png" in x.data)
    .map(x => image.decode(base64.decode(x.data.at("image/png")), ..args))
}
#let cell-name(cell) = {
  if not (cell.cell_type == "code" and cell.source.first().starts-with("##")) {
    return none
  }
  return cell.source.first().trim("##", at: start).trim()
}
#let parse-cell(json-data, name, image-width: 90%) = {
  let c = json-data.cells.find(c => cell-name(c) == name)
  let lines = c.source.slice(1).map(parse-line)
  let source = lines.map(array.first).join("\n")
  let bold = lines.map(x => "bold" in x.last())
  return (
    source: {
      show raw.line: it => if bold.at(it.number - 1) {
        text(weight: "bold", it)
      } else {
        it
      }
      raw(block: true, source)
    },
    images: parse-images(c, width: image-width).join()
  )
}

used like this:

In code_blocks.ipynb, define and execute a code cell with content

## setting-ranges
set xrange [-pi : pi] ## bold
plot sin(x)

and in the Typst document:

// Alias for parse-cell, pre-configured to work with our notebook
#let cell = parse-cell.with(json("code_blocks.ipynb"))

// Get source and outputs of cell with tag `setting-ranges`
#let setting-ranges = cell("setting-ranges")

== Setting Ranges

#grid(
  columns: (1fr, 1fr),
  [
    That was simple. Notice how gnuplot decided to plot
    our function from -10 to +10. That's the default,
    which we got because we didn't ask for any
    particular range. Gnuplot also set the y-axis
    limits (the range of the vertical axis) to
    encompass the range of the function over that
    default x-axis domain. Let's take control of the
    limits on the horizontal axis (the new command is
    highlighted). Gnuplot happens to know what π is
    (but doesn't know any other transcendental
    numbers).

    #setting-ranges.source
  ],
  setting-ranges.images,
)

Making/uploading/linking videos

The blog post mentions (just by description) a filter that creates animation frames, stiches them into a movie file, attaches the file to the PDF, copies the file to server and adds a link in the book.

For Typst, I would also do these things in the notebook, using the gnuplot kernel as an extension inside of a Python notebook and make the movie and server copy from there. The notebook cell that does this can output a link which is easy to pick up from the Typst side as shown above.

Conclusion

I’m rather biased since I love the Typst language, how it juggles with markup and script in the same file. But I think it’s fair to say that this kind of things is easier in Typst: you just need to learn Typst, rather than Pandoc Markdown, Pandoc filters and Panflute, maybe Lua, Python and Latex. And the language is optimized for this kind of things so the amount of boilerplate is much much less. Or at least I can say that I had fun writing the above Typst code, using the built-in support for JSON and YAML and a few glances at the Typst documentation, rather than a deep dive in the heterogeneous documentation of several large software projects.

It’s true however that Typst cannot do everything, for example calling external programs (except for some programs ported to function as WASM plugins). Though that particular limitation doesn’t bother me: already when using Quarto (which can run pandoc filters calling external programs) I had decided to keep code blocks in a separate notebook.

8 Likes

It’s a powerful, general-purpose programming language that has at least one very fast JIT compiled implementation. It’s popular as an embedded scripting language because it’s also incredibly small.

1 Like

Thanks for this detailed reply, which is useful for learning more about how to do things in Typst.

For me, I need to be able to translate a single source document into multiple formats (HTML, docx, PDF, etc.) so I need what Pandoc offers. (My article itself is an example of this: the version on my website is HTML, the version for the magazine was something else.)

The power of Pandoc filters is that they are not textual transformations, but operate on Pandoc’s internal representation. So, for example, if Typst continues to gain popularity, it’s probable that some day someone will write an output processor for Pandoc that emits Typst (this would be a Haskell program). When that becomes available, all my filters will automatically work for Typst. I will be able to take any of my documents and tell Pandoc to turn them into Typst, without knowing anything about Typst. And, if I want, I’d be able to get picky and tweak the Typst output by putting some extra code in my filters.

I rarely write directly in HTML or LaTeX directly any more, and I would certainly never write directly in Typst, lovely as it is. I write in my personal brand of Markdown, which is Pandoc’s Markdown extended with my own filters.

(Note that my article is a bit obsolete. The best way to write filters now for Pandoc is with Lua.)

2 Likes

I was under the impression that Pandoc can emit Typst since 3.1.2 from last year. For example, in 3.1.4.

1 Like

I totally agree on the usefulness of pandoc’s internal representation. Having an AST with explicit, semantic meaning for every element is a great power. But it’s a power based on restriction: you are limited to the extended Markdown syntax, and for everything custom you must resort to generic divs and spans with classes/attributes. This makes the syntax rather heavy. For me having a scripting language at hand in the markup, and the ability to define and call functions directly in the document, felt very empowering.

We’ll see how Typst develops regarding multiple output formats. HTML is a high priority for the developers now, and they also have epub in mind I think. I’m not sure about docx, but would guess that the work on HTML support would help a lot for docx too.

In summary I think Typst is firmly on the side of expressive source, while pandoc is on the side of restricted source with expressiveness provided separately in filters. This makes Typst more comfortable for my needs but probably there will be a loss when converting to some output formats.

2 Likes

I’ll be damned. Is it too late to go back in time and get credit for my prediction?

Interesting points. I’ll definitely be following Typst’s development. I’d seen references to it before but never paid much attention until your comments here.

3 Likes

Fletcher is based on CeTZ which recently changed licence to LGPLv3 - seems quite confusing to me