How can I start a Julia worker on my laptop from a remote server?

I can use ssh to connect to a remote server. Now I would like to display plots on my laptop (from the server) by launching a worker on my laptop using the addprocs function.

I have this code, which works fine as long as all Julia processes are on the same machine:

# Copyright (c) 2025 Uwe Fechner
# SPDX-License-Identifier: BSD-3-Clause

if Threads.nthreads() > 1
    function init_plotting()
        # Only add a worker if we don't have any dedicated worker processes
        if nprocs() < 2  # nprocs() counts main + workers, so < 2 means no dedicated workers
            println("No dedicated workers found, adding 1 worker...")
            @time addprocs(1)
            @eval @everywhere using ControlPlots  # Ensure ControlPlots is available on all workers
            @eval @everywhere using FLORIDyn      # Ensure FLORIDyn (including WindFarm) is available on all workers
            
            @everywhere function rmt_plot_measurements(wf, md, vis; separated)
                # Create a fresh plt instance just for this task
                local_plt = ControlPlots.plt
                return plotMeasurements(local_plt, wf, md, vis; separated=separated)
            end
            @everywhere function rmt_close_all()
                local_plt = ControlPlots.plt
                return local_plt.close("all")
            end         
        else
            println("Already have $(nprocs()-1) dedicated worker(s), total processes: $(nprocs())")
            println("Workers: $(workers())")
            
            # Ensure ControlPlots and FLORIDyn are loaded on existing workers
            @eval @everywhere using ControlPlots
            @eval @everywhere using FLORIDyn
        end
        nothing
    end
    init_plotting()  # This sets up workers and remote plotting capabilities   
end

Executing:

 @spawnat 2 Main.rmt_plot_measurements(wf, md, vis; separated=separated)

displays a plot.

But how can I start a Julia worker on my laptop, which has no publicly visible IP address by calling addprocs() on the server? Can I do some clever ssh tunneling?

1 Like

Is your laptop / server on the same network? I don’t know the answer to your question but perhaps can suggest an alternative solution. Your workflow was similar to mine: submit job to HPC (interactively) and then do analysis/plotting on the returned data on the head node. Initially, I used X11 forwarding to forward the plot to my local machine. This was great, but sort of cluncky. In the past few months however, I have been using sixel to display the plots in my terminal. This has worked amazing for me. I currently use Gnuplot.jl for plotting which has sixel support and I use Windows Terminal which also added sixel support last year.

1 Like

I’m not sure I quite fully understand the goal is, but if it’s just about creating workers on other machines, Distributed.jl has built-in support for that (if you didn’t already know)!

If you have the worker machine set up so you can connect via password-less ssh, then you can simply call addprocs([("RemoteHostName", nrp)]) where nrp is the number of worker processes you want to spawn on that host, and RemoteHostName e.g. ā€œElectromagnetismā€ is the name you gave the remote host in your .ssh/config file:

Host Gravity
    Hostname 192.168.X.XX
    IdentityFile ~/.ssh/keys/[PRIVATE_KEY]

Host Electromagnetism
    Hostname 192.168.0.XX
    IdentityFile ~/.ssh/keys/[PRIVATE_KEY]

Host StrongForce
    Hostname 192.168.0.XX
    IdentityFile ~/.ssh/keys/[PRIVATE_KEY]

Then everything should work exactly the same! That works great using the SSHManager for relatively simple clusters such as on a home LAN, but If you have a different cluster configuration or job scheduler, there are some curated for the most common ones such as Slurm and LSF.

The goal is, to run a simulation on a cluster, and to display the intermediate results live on my laptop. Similar to this video.
This was a simulation with 9 wind turbines, but I want to run simulations with 300 wind turbines, which I cannot do on my desktop.

I tried to ssh from the cluster into my laptop. So far it did not work. If both machines are in the university network, it might work somehow.

Mainly for the benefit of anyone else reading this thread, sandboxing the remotely spawned local worker environment is something that should be top of mind. But why not output the remote plots to its web server using something along the lines of Dash.jl, Genie.jl, or Pluto.jl?

Oh I understand now, thank you. My first, thought would be to mount the remote file system with sshfs, save the plot each update, and then use watch (assuming you’re on Linux) to open the image file periodically. Or, if the details of the cluster topology allow, you could have a cluster process call remote_do

remote_do(plot, LaptopProcessID, plot_args...; kwargs...)

to asynchronously execute the plot on the laptop. The laptop doesn’t send anything back either.

Good idea! I will try. I mean, there will be perhaps one or a few updates per second only.

Assuming the laptop is at home and the cluster is on a university/work network, how would this work?

Well, via VPN.

Something I did last year but didn’t get around to polishing enough for registration yet, but which might help you: GitHub - goerz/MuxDisplay.jl: A Julia display showing graphics in a multiplexer pane

See the YouTube video at http://www.youtube.com/watch?v=jvfhw2EI5i8 for an overview, or the README from PR #2

2 Likes

can you use GitHub - simonschoelly/KittyTerminalImages.jl: A package that allows Julia to display images in the kitty terminal editor somehow?

2 Likes

This really depends on the config, no? In my HPC experience, often it’s ssh into a head node and then use SLURM/PBS to allocate resources on the compute nodes. If I understand correctly, you want to use your laptop to essentially act as the head node and when you launch workers, you want these workers launched remotely. If you find a solution please let us know. This would greatly speed up my workflow as well.

Yet another option is using vscode with remote development. This way, for all intents and purposes, you are working on your ā€˜laptop’ and any plots that are produced can use vscode’s display engine.