Programming with rich text (Jupyter notebooks or similar?) on remote servers

Tooling
, ,
1

Hello,

I have been programming in Julia since 2020. I first used JupyterLab with IJulia.jl, and now I’m working with VSCode, still with IJulia.jl.

In this last case, I actually don’t like programming using the Julia extension and a pure Julia script, since I can’t write markdown text or equations, and I can’t keep the plots visible, even if I close and reopen it after months. Indeed, I usually code using Jupyter Notebooks.

I’m fine with that, but what I don’t like it that I have to install a python environment for that. I usually need to keep the simulation working on my remote machine while I turn off the ssh connection with VSCode. To do that, I need to start a manual Jupyter kernel under a tmux virtual terminal.

I was wondering if there exists a clever way to do that. I need something which is readable (a Julia script is not the case), which eventually keeps the plots displayed, which can remain in execution even if I disconnect to the remote server, and that eventually doesn’t require to installation of other packages like python. It is better if this can be run on VSCode, to take advantage of GitHub Copilot.

So, are there better options than Jupyter Notebooks?

2

what about

1 Like
3

Have you tried Pluto.jl? It is basically Jupyter but you don’t need a python environment. There is a big difference: in Jupyter, you run cells manually, whereas in Pluto the cells update automatically (changing one cell causes all dependent cells to recompute).

4

Yes I tried it a few times, but I have found any extension for VSCode. I really like VSCode, the way I can easily connect to remote servers, its git handling and its Copilot.

5

Pluto notebooks are valid Julia files, so technically, you could open the same notebook in Pluto and VSCode at the same time, use Pluto for basic development, and VSCode for Copilot and Git.

But I can see how inconvenient that could be. If VSCode is a hard requirement, then perhaps Pluto is not the right choice for you.

6

I’ ve implemented inline results and plot saving at Feat/persist inline results by xgdgsc · Pull Request #3123 · julia-vscode/julia-vscode (github.com) . But the extension devs weren’ t interested. It works with remote reconnections across client devices.

You can try Release A quick dot methods completion demo · xgdgsc/julia-vscode (github.com) . Should still work with 1.10. Last time I tried merging latest master from upstream to my branch and had many issues that I don’ t have time to fix. So it’ s based on a branch that’ s old.

And it also doesn’ t have markdown equation support.

2 Likes
7

You don’t. You can use the lightweight package NBInclude.jl to load a Jupyter notebook into a Julia script just as you would for include, no Python required.

(You can even use Jupyter notebooks as part of a Julia package/module in this way.)

8

But NBInclude is just for including notebook inside a Julia script. But it seems a bit strange, that I have first to write a Jupyter notebook, but then I run it inside a Julia script.

9

This seems like the jmd files where you mix markdown and Julia, right? But you have to render it, right?

10

You could run julia -e "using NBInclude; @nbinclude(\"my-script.ipynb\")".

Or write a 2-line wrapper shell script juliaipynb my-script.ipynb that does this for you.

(You could also use NBInclude to export the .ipynb file to a julia .jl file.)

The point is that NBInclude.jl takes care of the job of executing notebook files without requiring Python, but it is still up to you to invoke it somehow. The julia executable by itself doesn’t know about this package.

11

Is your issue basically this?

12

This doesn’t completely address all your issues, but I can tell you what I use as a workflow.

First, I think it is important to think of Jupyter as an application. It should never be part of your project environments: A kernel might be, but Jupyter itself is just some program (that happens to be implemented in Python) that you need to install somewhere on your system. Personally, I use a dedicated Miniforge environment. People get confused by stuff like the IJulia instructions to run notebook() in your Julia REPL. That might be good if you have a class to teach and you just need things to run, but it just creates a lot of confusion.

So, make sure you have Jupyter installed (on your remote workstation), with all the plugins you like. Then also install IJulia into your main Julia environment (not your project environments; luckily, Julia is better than Python in that respect). One plugin that’s absolutely essential (and that should be installed alongside Jupyter itself) is Jupytext. This allows you to link your .ipynb files to .md or .jl files that get synchronized automatically.

Now what I’ll do is connect to my remote workstation via ssh (with X-forwarding) and run tmux. I’ll open the .jl version of the notebook in neovim and a Julia REPL side-by side in two tmux windows. I’ll then use vim-slime to send code from the editor to the REPL. This is great for experimentation and to build up a notebook.

I’ll also have a Jupyter Lab server running on the remote workstation where I view the .ipynb version of the notebooks. I can access that server through SSH with port forwarding. As I said, Jupytext will make sure that these are synchronized (but you might have to do “reload from disk” sometimes). Once I finish experimenting, I’ll re-run the entire notebook (the .ipynb version) in Jupyter and save it. I might also work in the notebooks directly for smaller fixes, or if I’m comparing plots, or stuff like that. Those notebooks stay on disk, as something that I can show my boss if he drops in. I’ve also used Quarto to generate “reports” in PDF format from time to time.

So, basically, work in your editor and the REPL via ssh/tmux, and the .ipynb files automatically get saved as a permanent record of that work.

I’ll commit only the .md/.jl versions of the notebooks, not the .ipynb version, since they’re much more amenable to version control (the original motivation of juypytext!). The .ipynb version can also easily be recreated. Nowadays, I usually write myself a Makefile that regenerates any missing .ipynb files from a clean checkout.

The resulting repo might look something like this (this is research code, not really meant for public consumption). When a project ends, I’ll zip up the entire folder including all the .ipynb files with all the plots in them, for archival in addition to the git.

3 Likes
13

You may find VS Code Interactive Window the perfect tool for your needs.

You may support adding Julia as a featured language in:

I appreciate the pragmatic mainstream aimed design of the Julia eco system.
Hence I think it makes sense to support the JupytText format (Which is what Interactive Window implements).
I hope it will be added to Julia VS Code extension.

I find it the best format for literature programming as it gives all the capabilities of Jupyter in a version control friendly format.

1 Like
14

Thanks. Although I will mainly use VSCode instead of Neovim, the idea of working with .jl files and then automatically converting them into jupyter notebooks seems a good workaround.

I think that Julia lacks such a feature. I mean, Julia is built for numerical simulations and plots. It is supposed to have a native framework that allows you to do it seamlessly.

15

You want Julia to implement something like Jupyter notebook support in the base language? That seems inflexible to me.

I don’t understand why it is a problem to use a package like NBInclude, which you can execute from the command-line as I noted above.

(Julia doesn’t include plotting in the core language either. Neither does Python. And I’m glad it didn’t, because that allows plotting packages to evolve and diversify at a pace that the language cannot.)

16

Sorry, but I don’t get why NBInclude should be my option.

I still need to program using a framework that allows me to make the document commendable (latex and markdown) and readable like the jupyter notebook does. As far as I understood, NBInclude just imports the jupyter notebook code, but then I still program using a Julia file, which is not very readable and commendable.

How do you program using NBInclude? Do you first write the Jupyter Notebook, and then you include it inside a Julia file, and then you run functions in the Julia file? I don’t get it sorry.

BTW, I was wondering if the .jmd format would be an option for me. The Julia extension of VSCode supports it, and I can basically write a markdown file with the Julia code inside the code blocks. I still don’t understand if it is possible to render in live view (I change the code and the rendered output changes). Do you recommend it?

@Tortar mentioned Quarto, it seems very similar to the .jmd file extension. What is the difference?

17

No, you can program entirely using Jupyter notebook files if that is what you want: they just become your source file. You don’t need to create any .jl file as I explained above: you can execute your notebook directly from the command line.

That is:

  1. Create/edit a my-script.ipynb notebook file using whatever Jupyter-compatible program that you like (Jupyter, JupyterLab, VSCode, …). This contains all your code, and whatever additional rich text you want.
  2. When you want to run it outside of the notebook environment, e.g. on a remote server, just do julia -e 'using NBInclude; @nbinclude("my-script.ipynb")' — no Python or Jupyter installation required.

The only time you would need a .jl file would be if you want to make a Julia package out of your code, and even then you only need a 4-line Foo.jl file

module Foo
using NBInclude
@nbinclude("foo.ipynb")
end

and the rest of your code can be in Jupyter notebooks if that is what you want.

18

But ten you loose the concept of interactive coding, with cells to run etc, right?

19

No, you can code interactively in the notebook environment as normal. But you can also save the notebook and run it non-interactively with NBInclude, without having to convert back and forth between .jl files and notebooks.

1 Like
20

Ok, I see. But then I still keep coding with Jupyter notebooks, which is what I’m already doing. Good to know that I can simply include it using this package.