Notebook Use Case for Mathematicians?

Tooling
, ,
1

TL; DR: Jupyter, Pluto, Neptune, Weave, and LaTeX, Julia via LuaLaTeX, or Julia via Minted or Listings in LaTeX, all don’t satisfy my use case as a mathematician: I’d like top-down cell evaluation with reactivity and interactivity applied to both Julia programming and LaTeX macros and expressions.

Having caught up with the discussions on the use cases and purposes of Jupyter, Pluto, and particularly Neptune, I thought I might throw my hat into the ring. Hopefully the length of this post is alone enough to demonstrate how big of a pain point my suggested solution addresses.

As an applied mathematician and programmer, I’m wondering if anyone else would benefit as I would from a notebook package/functionality with top-down linear evaluation order like LaTeX and Weave, which would allow reactive LaTeX macro updating for example.

This single but all-effecting feature allows complicated and long mathematical expressions come to life, with the Julia implementation following the mathematical expressions/derivation. Like lecture notes or documentation for a large mathematical software implementation.

Let’s face it: in the proliferation of tutorials and presentations using Jupyter or Pluto notebooks, they are all read and presented from the top of the notebook, downward. I see Pluto’s direction to be more towards being an Excel + GUI-producing service, more appropriate to be exported as an interactive web page than a mathematically detailed PDF document…

…which is actually what I make in my profession and free-time: applied mathematics documentation (lecture notes, study notes, mathematical reports, etc). When I use Pluto (which is the closest for my use cases) I feel like I’m using a screwdriver (Pluto) to drive a nail (my use case). It gets the job done more easily than, say, a wrench (Jupyter, Weave).

I am not presenting my thoughts to criticise Pluto - far from it, in fact. For what it is, I love Pluto. For the large majority of my use cases, it’s an awkward but only mostly doable fit and I’ve impressed multiple colleagues who use MATLAB or C++ with it.

What I’m trying to say is: if my documentation task is more mathematical than programmatical, I use LaTeX. If my doc task is more programmatical than mathematical, I use Weave or Pluto. If my doc task is more interactive visualisation then I definitely use Pluto. If I want Jupyter-like notebook presentation with the benefits of being a Julia script, I’ll use Neptune (which I still don’t fully understand the goal of). But if I want all of the above functionalities of interactive mathematical implementation? I’m stuck with the trade-offs each option gives.

Eventually when I’m confident enough with Julia programming and package development, I may fork Pluto into a variation that evaluates top-down linearly, modifying for LaTeX reactivity as well. (Hint-hint, I’d be keen to support someone else leading such a project, if it meets enough people’s use cases.) I might call it Zeus because variables and functions are defined once (like in Pluto), but can get called in multiple other places visually after it’s defined (unlike Pluto), like forked lightning.

I consider myself in the limbo between novice and professional programming, and am not too familiar with the internals of each of the notebook services mentioned above. I’d love to hear everyone’s opinions on what I’ve shared, and please feel free to let me know if I have any logical or other errors in my thoughts.

Edit: To add to the list of solutions, I also know that one can run Julia alongside LaTeX and get the output presented in the output doc (via any of the two options, one via Lua, and another via listings or minted I think it was), but again with the two(-or-more)-language problem: Pluto does math (weakly) and programming, and is easier to use.

Edit 2: @ess3sq explains that a top-down-evaluating version of Pluto may not provide LaTeX reactivity.

7 Likes
2

I‘m not sure whether I understand correctly. You want a system where you can generate documents containing Julia code, Julia output and LaTeX equations all in one document? Something like R Bookdown? But then with the reactivity of Pluto, that is, change code and see the output change, within about a second, on a webpage?

Edit: whatever system one makes. There will always be trade offs, so I‘m a bit confused by that part.

1 Like
3

Ah, thanks for asking for clarification.

I think the best way to summarise it is, Pluto but top-down evaluation, with attention to LaTeX macros as well.

LaTeX macros only kind of work in Pluto as it is, but macros won’t update reactively. For example,

image
image897×523 14.7 KB

then I’ll change the definition of \Par to use square brackets instead of parentheses, and

image
image896×525 14.3 KB

the second cell doesn’t update.

I guess I’m looking for the equivalent of Julia addressing the multiple language problem. As a crude summary, Julia challenges for having the speed of C and legibility of Python. I’m looking for a notebook service that provides the mathematical presentation of LaTeX and the programming capabilities of Julia, as a Julia package like Pluto.

I’m pitching this top-down linear cell evaluation because in theory it would solve the above demonstrated non-reactive LaTeX macro updating lack of functionality.

2 Likes
4

And do you want to show the LaTeX macro to the reader or is it just for your own use? Why not give the macros unique names instead of redefining them?

EDIT:

Aha, now I get it, I think. Basically, Pluto’s creation of the DAG for evaluation should take LaTeX macros into account because there is now hidden state.

5

Me personally, I wouldn’t show any LaTeX code to the reader. Definitely my own use.

Oh, yeah I just used \Par as an example. I could’ve used \xbf for boldface x or something, I tend to use Par for parentheses, it makes my LaTeX code easier to read. LaTeX distinguishes \par from \Par.

I’m going to assume DAG is “directed acyclic graph” for the order of cell evaluation? If so, not really because that would be complicated to deal with two partitions of objects overlapping by the cells they’re defined and called in (not too sure how to describe this). Hence why I’m pitching: just evaluate each cell in order. For faster development, maybe implement DAG for LaTeX macros and Julia objects, but I’d be content with top-down.

6

You can use Julia to define your aliases, then cell updates would work correctly.
E.g.

par(x) = "\left[ $x \right]"

md"""
`$(par("x^2"))`
"""
6 Likes
7

This sounds like the discussion around Neptune. From the Neptune README:

Neptune originated as a fork of the Notebook platform Pluto, for those (like Data Scientists) who have requirements which conflict with the ‘reactive’ aspect of Pluto (where all cells re-evaluate every time a single cell is evaluated or re-evaluated), and instead prefer linear, sequential execution.

1 Like
8

This I’m aware of as well, and thank you for bringing it up, I think it does solve the reactivity problem.

However, when typing math expressions I don’t want to be jumping between Julia’s $(...) mode and into Julia’s "..." string mode and md"..." markdown mode just to write an equation, let alone a derivation.

9

Yes, I did read their README, and I was excited about it. But when I tried it out, I got:

image
image687×635 15.2 KB

which happens to behave more like a Jupyter notebook, i.e. the existing value of a variable depends on the last executed cell that defined it. Which is not what I’m looking for.

I like Pluto’s, “once you save a cell, it runs it and checks for conflicting definitions, etc.”

From Neptune’s README,

and instead prefer linear , sequential execution.

which I don’t understand if it hasn’t been incorporated yet, or they have an interpretation of “linear” and “sequential” that conflicts with mine.

10

I think you just need to overwrite the topological order calculation in Pluto to get such a behavior:

1 Like
11

That makes a lot of sense.

I’d have to look into a way to override Pluto’s definition of topological_order for every version, in the scenario that my use case isn’t of interest to enough people in the package development “business”.

Otherwise I could submit a pull request for a simple “button” in the browser interface for Pluto that would switch to a linear version of topological_order, but by the sounds of discussions about Neptune.jl, they might not accept it.

P.S. love the xkcd reference they have in the code.

12

I 100% agree with the premises of the original post and the concerns voiced there because I experience the same hurdles when using Pluto, although I should specify that Pluto is still way better than the competition.

I have also personally experienced this issue with updating markdown in other cells, but I don’t think what is suggested in the quote above is a viable solution to this problem.

Indeed, it would update the LaTeX formulae if markdown cells were all always executed after one of them has been updated. This is not possible because Pluto only updates cells that were affected by the changes, and since unlike Jupiter Notebook a markdown cell is no different than a Julia code cell, all Pluto sees is some string literal being updated which is not being used in any other piece of Julia code.

Now there are two “obvious” solutions (“obvious” in the sense that they come to mind but are not actually viable):

  1. Execute all cells after a change was made. This would fix LaTeX, but is not viable because if intense calculations are made, you don’t want to have the machine perform them again just because you updated some formula.

  2. Change the way Pluto works to “remember” which are markdown cells and which are not, and if a markdown cell is updated, all other markdown cells are updated. This could be helpful if one assumes that latex cells can be evaluated quickly because they don’t contain actual heavy calculations, but that fact is just not true because you can have calculations in a markdown cell if you employ the use of $ string interpolation or wrap the contents of the cell in a begin... end block and add some more code.

Another problem is that Pluto relies on MathJax to do the LatEx rendering as I have personally discovered recently and has little to no control over fixing MathJax’ limitations.

TLDR; I don’t think top-down evaluation would fix LaTeX/markdown issues.

// EDIT
Clarification: top-down would improve my own experience using Pluto in a couple of ways, but not the LaTeX issues.

1 Like
13

The lack of viability of the two options you’ve suggested makes sense, thank you.

This reverts my original question to simply, if one was to start from scratch, how would one design a notebook service that provides interactivity and reactivity for both LaTeX rending and Julia programming, and would such a service be a better notebook option for anyone. I’ll update my original post.

This would also help with many of my use cases that don’t involve LaTeX, so I agree with you there. Additionally, Neptune.jl shoots off to the other extreme for me when it comes to functionality, as vaguely described in one of my previous responses in this thread.

Agreed, for more programming-heavy documents. For math documents, I’ll be sticking to LaTeX itself. And I’m stuck for when I want both, a two-language problem on its own, if you will.

14

An other way to execute the markdown / Latex cells in the required order is to explicitly define the dependencies for Pluto using dummy Julia variables:

begin
nc1 = nothing
md"""
...
\newcommand\Par...
"""
end 

begin
nc1
md"""
\Par
"""
end

This way, the cell dependencies in Pluto are set as you want, the 2nd cell is always executed if the 1st one is updated.

4 Likes
15

If what you want is lecture notes and documentation with a linear evaluation order with LaTeX and etc, perhaps what you want is just Julia’s Weave.jl?

Or maybe you can explain what it is that Weave doesn’t do that you want it to do? If it’s the interactivity, I don’t think you can really get linear order evaluation and reactivity efficiently. In essence you’d have to call “weave” every time you made any changes to anything, and it’d be slow and cumbersome. The topological sort of Pluto ensures that you only update the stuff that changed, I think.

16

Open an issue on the github repo, the pluto people (plutonians?) are very helpful so maybe they can assist you.

17

The pluto team is working on making meta-programming reactive. This means that you can automate the generation of proxy variables and make it invisible to the user.

I made a quick (and very broken) prototype using the branch on Analyse macros by Pangoraw · Pull Request #1032 · fonsp/Pluto.jl · GitHub. I used the notebook which uses Katex to embed objects in LaTeX as a starting point.

Basically I made the tex macro emit variable assignments and usage for the commands used. The reactivity engine is able to analyze them and provides reactivity.

reactive_latex

I like that with meta-programming, you get the tools to automate these sort of things without touching the Pluto code itself. The same is also true with Julia where a lot of construct are implemented as macros where a reserved keyword would have been required in another language (ex: @assert).

Here is the sample notebook: Katex hyper fun · GitHub (Let me know if you have any issue)

10 Likes
19

Yes, you’re exactly right. It’s nice that we have the Julia reactivity and interactivity.

So much of my studying is exploring mathematics, with some theory for derivation, exercises, etc, then implement them in Julia to see it come to life.

No platform provides an efficient way to explore both at the same time, that’s what I’m wondering if people would benefit from. Thanks for asking!

20

The idea for varying Pluto’s notebook execution or diversifying its functionality related to what I’ve prompted for discussion here has been brought up multiple times with the Plutonians (nice naming haha) in various discussions, including one I’ve had previously (which surprisingly, I can’t find, maybe they deleted it). And I’m completely supportive of the argument that as great as Pluto is, it just doesn’t cover everyone’s use cases. Hence why I’ve started this topic. They seem quite immovable in their end-vision for Pluto, arguably they’re even unhappy about Neptune.jl, which doesn’t make sense to me.

21

Not a fan of putting double backslashes everywhere, but it looks promising!

Off topic a little, better handling of macros in Pluto would be amazing. And it looks like it will have nice implications for math expressions in Pluto. I’ve subscribed to the PR Conversation to stay posted on its progress.