I would like to announce MuxDisplay, which I just submitted for registration. It is a package that I wrote a while ago to fulfill my own needs for displaying graphics (plots, mostly) when working in the REPL, especially on remote systems via SSH. I’ve previously mentioned it in passing in How can I start a Julia worker on my laptop from a remote server? - #10 by goerz. Using multiplexers as a “canvas” is something that I’ve found improves significantly on the various approaches for “inline plotting” discussed in
If you’re so inclined, there is an introduction to the package on YouTube:
The above thumbnail shows how using MuxDisplay will generally look like in the context of tmux panes, with the top-right panel being used for plotting output from the REPL.
From the package README:
Suppose you are working remotely from your MacBook on a Linux workstation, accessible via SSH. You want to keep a record of your work, so you are running a JupyterLab instance. You could access that Jupyter server on your laptop through port forwarding, but that has all the drawbacks of a web interface. So, you use the Jupytext extension to link your .ipynb files to .jl (or .md) files that automatically stay in sync, and edit those .jl files with Neovim inside a tmux session running on the workstation. Now, in order to actually run the code, you open a Julia REPL in a tmux split-pane, and use the vim-slime plugin to send snippets of code from the .jl version of the notebook to the REPL.
Things get a little tricky once the code generates graphics, e.g., via Plots.jl. One possibility is to use SSH X-forwarding. But, this requires having to install an X-server on your MacBook, and depends on a fast network. Plus (and this applies when plotting locally from the REPL as well), you only get one plot at a time. Another possibility is the amazing UnicodePlots.jl. Works great, but has obvious limitations. Now, modern terminals like WezTerm, iTerm2, and Kitty actually support showing high-resolution graphics with their own iTerm inline graphics protocol, the Kitty terminal graphics protocol, or the standardized (but rather inefficient) Sixel protocol that is supported by quite a number of terminals.
There are packages like ITerm2Images, KittyTerminalImages, and SixelTerm that hook into Julia’s multimedia display system to automatically show inline graphics. These work extremely well if the REPL runs in a terminal like WezTerm (which supports all three image protocols). Sizing can be an issue. KittyTerminalImages has a nice option to control the size of the images.
However, we are working inside tmux, and this throws a monkey wrench into things. Inline graphics protocols require tmux to support OSC “passthrough”, which is available as of tmux 3.3 and requires set -gq allow-passthrough on in your .tmux.conf file. The program emitting the escape sequences for image display must be aware that it is running inside tmux and must modify its output accordingly. Programs like wezterm imgcat and iTerm’s imgcat do this, but, e.g., KittyTerminalImages does not. Sixel support requires tmux 3.4 and tmux must be built with --enable-sixel (which, e.g., brew install tmux on macOS has enabled). Of course, you should also make sure tmux is generally set up correctly, see the FAQ. Even with image support inside tmux, if you are using panes (horizontal or vertical splits within the same window), images often end up in the wrong place. This seems to heavily depend on your terminal emulator, and can ruin “inline” images. Scrolling inside tmux or switching between tabs (“windows”) will always make the graphics disappear.
Instead of throwing up our hands, we might as well lean into our multiplexer. Instead of inline graphics, we’ll use a dedicated pane to show images. This is what the MuxDisplay package provides, hooking into Julia’s display pipelines in a similar way as KittyTerminalImages, etc. Instead of directly emitting the image, it sends instructions to the multiplexer to execute an external image viewer (like imgcat) in a specific pane. With tmux, this still requires passthrough to be set up correctly, but it gets around many of the practical issues with image display in tmux.
Screencast of MuxDisplay in a remote Tmux.pane:
- On the remote workstation, start a tmux session. Split the window into panes, with Neovim running on the left, and the Julia REPL on the right, with another pane above it for plotting (pane index
1, cf. theC-b qshortcut) - Make sure to have an
imgcatprogram in yourPATHon the workstation and thatMuxDisplayis installed in your base Julia environment - In the REPL, run
using MuxDisplay; MuxDisplay.enable(target_pane="1") - Issue plot commands from the REPL (e.g., from the Neovim pane on the left via
vims-slime)
The plots will show up in the top right pane.
If you have issues with the images not being properly placed in the target pane, either get a better terminal emulator, or use a separate dedicated tmux session (session name, e.g., Plots) that you can open in a separate window. You would then connect to that session with MuxDisplay.enable(target_pane="Plots:0.0"). For an even smoother experience, or if your tmux is too old (or you can’t figure out how to set up passthrough), consider using the WezTerm multiplexing feature to open a remote WezTerm pane that can be used for plotting.
Screencast of MuxDisplay in a remote WezTerm window:
- On the remote workstation, start a tmux session. Split the window into panes, with Neovim running on the left, and the Julia REPL on the right. There is no plotting pane this time.
- Make sure to have the
wezcatprogram in yourPATHon the workstation, in a version that matches your local WezTerm terminal. This is necessary both for the multiplexer and to provide thewezterm imgcatprogram. - In your local WezTerm terminal (which you’ve used to ssh into the workstation), open a second window with a tab in the
SSHMUX:workstationdomain. Runprintenv | grep WEZTERM_PANEthere to figure out the (remote) pane ID. Let’s say it’s5. - In the REPL, run
using MuxDisplay; MuxDisplay.enable(multiplexer=:wezterm, target_pane="5") - As before, issue plot commands from the REPL (e.g., from the Neovim pane on the left via
vims-slime)
Note that we are still using tmux as our main multiplexer, since WezTerm doesn’t have an exact match for the concept of tmux sessions. Of course, if you don’t generally use multiple tmux sessions on your remote workstation, you could just replace tmux with the WezTerm multiplexer, and your life will be much easier (the solutions for inline plotting will actually work out of the box).
In our little scenario, suppose that you have some conference travel later in the week, where you want to continue working on your remote tmux session. You’re not checking luggage, so you are only bringing your iPad for the trip. To connect to your workstation, you use the Blink Shell iOS app (probably the best-engineered and feature-rich iOS app ever, despite its unassuming appearance). We won’t be able to use the WezTerm multiplexer. Blink supports the iTerm inline graphics protocol but seems to have a lot of issues with image placement. It works okay with a dedicated Plots tmux session and a little tweaking. You will need to have the iTerm imgcat script in your PATH on the workstation.
Screencast of MuxDisplay from an iPad with Blink.sh and Tmux:
- Connect to your remote tmux session (with two panes, Neovim on the left and the Julia REPL on the right) in Blink
- On your iPad, open a second Blink window in slide-over mode
- In the slide-over window, also connect to the workstation, and start a separate tmux session with
tmux new-session -s Plots. - In the full-screen Blink window, in the Julia REPL, run
MuxDisplay.enable(target_pane="Plots:0.0", nrows=3, clear=false)
The nrows determines the height of the output (three plots in the tall slide-over window). With clear=false and the peculiarities of how tmux interacts with Blink, this creates a nice scroll-effect. It may take a few plots for the slide-over window to find its groove . If the slide-over window is too small, you can put it in a proper split view instead.
As I said, the package has been mostly developed to fulfill my own needs, and it’s definitely a bit on the “hacky” side. Still, I’d love for people to try it out, give feedback, and maybe contribute support for other multiplexers or shells.