[ANN] Native Julia engine for quarto using QuartoNotebookRunner.jl

We have internally converted a lot of Pluto notebooks to quarto, that mostly went without a lot of manual intervention I think. You don’t need to go through pandoc

Nice, could you (maybe in a new thread or so) provide a bit more detail how to do that?

Las November I spend almost a week trying to make a Quarto version of the GMT manual. Although nice to start with, I found it very very slow to run many scripts that run Julia code. But the worst was the cross-reference mechanism between files. Remember that I had to give lots of absolute paths. Was such a pain that made me give up.

The new engine will not help with general gripes one might have with quarto, we just do the execution of Julia code

1 Like

Yep, I guessed that much. My main grip with Quarto is its cross-referencing system.

Can anyone link me (or provide of the top of their head) a brief comparison between using Quarto and just standard Jupyter notebooks in VSCode directly (without the need to open a Jupyter server on your internet explorer)? Are there many arguments to give up Jupyter in favor of Quarto?

(kinda off-topic, i understand, hence searching for a link!)

In brief (and in my opinion), Jupyter notebooks are for exploration and keeping notes of findings, while Quarto is more geared towards authoring documents with code interweaved.

I wrote my thesis in Rmarkdown (now succeeded by Quarto) and was able to produce a PDF that met my school’s formatting requirements. I don’t think I could do that with Jupyter.

Edit: Quarto also let’s you seemlessly mix code from different languages in one document. Quite useful at times :slight_smile:

2 Likes

You can render preexecuted jupyter notebooks with quarto, so it kind of sits atop that workflow. You can also have quarto execute the code in there, but in that case it’s more common to just use the qmd format directly.

1 Like

Just to note: at the moment QuartoNotebookRunner.jl doesn’t support multi-language notebooks. It’s likely something we’ll eventually get around to supporting, since we have discussed it internally, but so far it hasn’t been the top priority. We’d most likely just hook into RCall.jl and PyCall.jl / PythonCall.jl for evaluation of those cells.

4 Likes

A prerelease has been published Release v1.5.29 · quarto-dev/quarto-cli · GitHub

9 Likes

How did I break it already?(?) :man_facepalming: :slight_smile:

foo.qmd

---
title: Foo
engine: julia
---
/Users/jsa % quarto --version
1.5.29

/Users/jsa % quarto render foo.qmd
ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: /Users/jsa/foo.qmd
ERROR: IOError: pwd(): no such file or directory (ENOENT)
Stacktrace:
  [1] uv_error
    @ ./libuv.jl:100 [inlined]
  [2] pwd()
    @ Base.Filesystem ./file.jl:63
  [3] evaluate_raw_cells!(f::QuartoNotebookRunner.File, chunks::Vector{Any}, options::Dict{String, Any}; showprogress::Bool, chunk_callback::QuartoNotebookRunner.var"#chunk_callback#61"{TCPSocket})
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/Mx5PG/src/server.jl:574

Not sure :slight_smile: wasn’t able to reproduce that one myself. Would you be able to open an issue with the details and hopefully we’ll be able to reproduce it somehow.

1 Like

I spent quite some time setting up a quarto notebook with Julia, Python and R and I used RCall.jl and PythonCall.jl to run the python and R code cells. This works reasonably well with the jupyter engine but with QuartoNotebookRunner it doesn’t show the plots from R or Python. I can open an issue if that is wanted.

If you start working more on multilanguage support , please ping me.

Sure, please do open one up @Fliks.

Kudos! That’s awesome.

Is reactive execution and/or an ability to selectively re-evaluate a single cell something that’d be on the roadmap?

I think neither of these are even part of the current (Jupyter-based) workflow.
Maybe that is more a focus for Pluto?

In the VS Code extension of Quarto you can run a life server that re-renders the notebook once you safe it, but I think Quarto itself is not really meant to be selectively re-evaluating.

The Quarto documentation does claim to allow selectively re-running individual cells when using Python in the VS Code Quarto extension, but for some reason the last time I tried (recently) on my work machine, there was no “Run cell” button available. (The screenshots show a “Run cell” button above each cell in the .qmd file.)

You are right, that also works with the VS Code extension and Julia,
but that sends the code to REPL, I think.

A quarto qmd file is really just a mixture of markdown and code with some advanced styling related metadata that is only relevant for the different output modes. There are no “cells” like in Pluto or Jupyter where you get output cells beneath or above them while interactively coding, there are just code blocks which you can execute in your repl either with a button or with alt+enter.

As mentioned above, there’s a preview mode where at each save of the document, a new preview is rendered and shown in a browser tab or in a VSCode pane. Of course this is only practical if your code runs reasonably fast, lets say on the order of a couple seconds. As QuartoNotebookRunner keeps a session alive, you only wait for package loading the first time you run your code. If there are other things that take time, I’d argue they should either be computed outside of the notebook, or you’d use something like DrWatson’s produce_or_load to cache results after the first run.

So quarto is not the right tool for exploratory analysis, its strength is creating good looking outputs in a variety of formats. As mentioned somewhere above, a good idea could be writing a small conversion package from Pluto notebooks, so that you can switch over to quarto once you are reasonably happy with your code. I’m personally completely fine with doing the exploration by executing code blocks in the REPL, and that works completely fine when using the qmd format.

4 Likes