How to run julia notebooks on cluster computers?

New to Julia
1

Hi! I write most of the code in Jupyter notebooks or in Pluto notebooks, however recently I have been asked to shift to cluster computers, However I am unaware of how exactly or even if it is possible to run code written in notebook format on such clusters, I know how to do this for compiled languages like C/C++ but not for Julia. Could anyone help me out ?

2

One solution is to use Pluto.jl. Start by ssh-ing into your cluster manager, then request a dynamic session with resources sufficient for what you’ll need. Start a Julia session once that’s done, then install and run with using Pluto; Pluto.run(). This will create a notebook session you can connect to (default address being http://localhost:1234/....). Then create an ssh tunnel between that address and your own PC, and you should be able to visit the address Pluto generated as a live notebook running on the cluster.

3

You can start Pluto in the cluster frontend node and prevent starting a browser:

julia> using Pluto
julia> Pluto.run(launch_browser=false)

Then use ssh to forward the port of the notebook to your local host, something like this:

ssh -4Y -L 1234:localhost:1234 yourcluster
1 Like
4

Thanks for your help I’ll try them and let you know how it goes. Thanks again

Rajarsi

5

Hi,

The last command is not clear for me. On the cluster, I ran
julia> using Pluto
julia> Pluto.run(launch_browser=false)
It gives a link, and if I copy and paste it in my browser, the notebook will be started on my local computer not on the cluster. I wanted to run Pluto notebook on the cluster.

Best,
Mohammad

6

Hi Mohammad, welcome to the Julia community!

The last command is important: it transfers the cluster’s Pluto port to the local computer. It works in Linux and probably in Mac OS but perhaps not like that in Windows. Which OS are you running locally?

7

Thanks for the replay,
I am working on Windows, and the cluster is running on the Linux.

8

Which ssh client are you using? Basically the concept is that we need to use SSH to create a TCP tunnel to your cluster node. It’s kind of like a VPN but to a very specific computer on a specific port.

Windows 10 comes with openssh as an optional feature:

You can then use the ssh command in cmd terminal or in Windows Terminal (highly recommended).

On Windows, I sometimes use putty. In this case, you would need to add a port forward as in this screenshot.

image

Depending on your cluster configuration, you may need to issue two SSH commands: 1) one ssh to your login node, 2) another ssh from the login node to the compute node. I recommend contacting your cluster admin for help with this. Alternatively, we may be able to help if you could point us to documentation for your specific cluster configuration.

1 Like
9

Many thanks for the answer.
I am using Putty for ssh client. Your method worked for me. I have one ssh to login the Cluster, and I made another ssh as you said in the previous post.