I can use ssh to connect to a remote server. Now I would like to display plots on my laptop (from the server) by launching a worker on my laptop using the addprocs function.
I have this code, which works fine as long as all Julia processes are on the same machine:
# Copyright (c) 2025 Uwe Fechner
# SPDX-License-Identifier: BSD-3-Clause
if Threads.nthreads() > 1
function init_plotting()
# Only add a worker if we don't have any dedicated worker processes
if nprocs() < 2 # nprocs() counts main + workers, so < 2 means no dedicated workers
println("No dedicated workers found, adding 1 worker...")
@time addprocs(1)
@eval @everywhere using ControlPlots # Ensure ControlPlots is available on all workers
@eval @everywhere using FLORIDyn # Ensure FLORIDyn (including WindFarm) is available on all workers
@everywhere function rmt_plot_measurements(wf, md, vis; separated)
# Create a fresh plt instance just for this task
local_plt = ControlPlots.plt
return plotMeasurements(local_plt, wf, md, vis; separated=separated)
end
@everywhere function rmt_close_all()
local_plt = ControlPlots.plt
return local_plt.close("all")
end
else
println("Already have $(nprocs()-1) dedicated worker(s), total processes: $(nprocs())")
println("Workers: $(workers())")
# Ensure ControlPlots and FLORIDyn are loaded on existing workers
@eval @everywhere using ControlPlots
@eval @everywhere using FLORIDyn
end
nothing
end
init_plotting() # This sets up workers and remote plotting capabilities
end
But how can I start a Julia worker on my laptop, which has no publicly visible IP address by calling addprocs() on the server? Can I do some clever ssh tunneling?
Is your laptop / server on the same network? I donāt know the answer to your question but perhaps can suggest an alternative solution. Your workflow was similar to mine: submit job to HPC (interactively) and then do analysis/plotting on the returned data on the head node. Initially, I used X11 forwarding to forward the plot to my local machine. This was great, but sort of cluncky. In the past few months however, I have been using sixel to display the plots in my terminal. This has worked amazing for me. I currently use Gnuplot.jl for plotting which has sixel support and I use Windows Terminal which also added sixel support last year.
Iām not sure I quite fully understand the goal is, but if itās just about creating workers on other machines, Distributed.jl has built-in support for that (if you didnāt already know)!
If you have the worker machine set up so you can connect via password-less ssh, then you can simply call addprocs([("RemoteHostName", nrp)]) where nrp is the number of worker processes you want to spawn on that host, and RemoteHostName e.g. āElectromagnetismā is the name you gave the remote host in your .ssh/config file:
Then everything should work exactly the same! That works great using the SSHManager for relatively simple clusters such as on a home LAN, but If you have a different cluster configuration or job scheduler, there are some curated for the most common ones such as Slurm and LSF.
The goal is, to run a simulation on a cluster, and to display the intermediate results live on my laptop. Similar to this video.
This was a simulation with 9 wind turbines, but I want to run simulations with 300 wind turbines, which I cannot do on my desktop.
I tried to ssh from the cluster into my laptop. So far it did not work. If both machines are in the university network, it might work somehow.
Mainly for the benefit of anyone else reading this thread, sandboxing the remotely spawned local worker environment is something that should be top of mind. But why not output the remote plots to its web server using something along the lines of Dash.jl, Genie.jl, or Pluto.jl?
Oh I understand now, thank you. My first, thought would be to mount the remote file system with sshfs, save the plot each update, and then use watch (assuming youāre on Linux) to open the image file periodically. Or, if the details of the cluster topology allow, you could have a cluster process call remote_do
This really depends on the config, no? In my HPC experience, often itās ssh into a head node and then use SLURM/PBS to allocate resources on the compute nodes. If I understand correctly, you want to use your laptop to essentially act as the head node and when you launch workers, you want these workers launched remotely. If you find a solution please let us know. This would greatly speed up my workflow as well.
Yet another option is using vscode with remote development. This way, for all intents and purposes, you are working on your ālaptopā and any plots that are produced can use vscodeās display engine.
Iād love to have a good solution to this problem. Iām afraid all of us working with HPC cluster experience similar problems and find some hacky solutions to it. For example I often produce terabytes of data which I have to visualizeā¦ most often than not the rapid solution is compress them in some way, download and visualize. I took a look at systems like paralook and similarā¦ these do work for some things but very terribly for other things.
The best solution I found which works for me is to run a jupyter notebook on the server and construct the visualization system I need in there manually. It is a pain in the ass, but often it is quicker to write the visualization software than to wait for results and download everything. I do not know if this applies to your use case.
What I generally do is start a job who computes what I need and another job running just the jupyter notebook with little allocated resources. If you need interactivity you may start the jupyter notebook with all required resources and in there run your functions. At the start of your job choose a password and a port and log to some file the node in which the notebook is running. Once the job starts copy that and do a reverse ssh tunnel from your laptop ā head node ā compute node.
Open your web browser on that port and run what you need to run.
Yes, this process is a huge pain in the ass and I did not find a way yet to automatically generate the ssh tunnel, you have to do it manually every time. But wellā¦ better than waiting 24 hours for data to download.