Best practices to share simulations with collaborators?

Tooling
18

I am attaching my complete but messy notes for the installation of our lab Ubuntu 20.04 server and the apache config file. Note there is much more there than what you need, as it is a computational server with JupyterHub, RServer Studio and a file server with NextCloud. But I am attaching it as there are some important parts for dealing with security (eg. setting up a basic firewall) and installing the middleware in general (sections 12 and 13).
What you need for JupyterHub and Julia kernel specifically is this:

# ---------------------------------------------------
# 16 - Install Jupyter and Jupyter Hub

sudo apt-get install npm nodejs-legacy
sudo apt-get install python3-pip
pip3 install --upgrade pip
pip3 install jupyter
npm install -g configurable-http-proxy
pip3 install jupyterhub
pip3 install --upgrade notebook
cd /etc
jupyterhub --generate-config
nano jupyterhub_config.py

c.JupyterHub.base_url = '/jupyter/'
c.JupyterHub.logo_file = '/var/www/example/imgs/jupyter-lef-logo.png'

# Enable jupyterhub as a system service:
nano /lib/systemd/system/jupyterhub.service
nano /etc/systemd/system/jupyterhub.service
(edit both)
>>>>
[Unit]
Description=Jupyterhub

[Service]
User=root
Environment="PATH=/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
ExecStart=/usr/local/bin/jupyterhub -f /etc/jupyterhub_config.py

[Install]
WantedBy=multi-user.target
<<<<
sudo systemctl daemon-reload

# enable at startup:
systemctl enable jupyterhub.service

To start jupyterhub:
- manually (for test): jupyterhub
- as a service: sudo systemctl start jupyterhub (also: stop|status)
If you stop it, then run also `sudo pkill node` or jupyterhub will not restart as it will find the proxy port busy.

# Autostop idle kernels AND servers..
Download cull_idle_servers.py from https://github.com/jupyterhub/jupyterhub/blob/master/examples/cull-idle/cull_idle_servers.py
and put it in /etc/jupyter

python -m pip install tornado

nano /etc/jupyterhub_config.py
>>>>>>>>>
c.JupyterHub.services = [
    {
        'name': 'cull-idle',
        'admin': True,
        'command': 'python /etc/jupyter/cull_idle_servers.py --timeout=604800'.split(),
    }
]
<<<<<<<<<<
(1 week)

cd /etc/jupyter
cp /root/.jupyter/jupyter_notebook_config.py .
nano /etc/jupyter/jupyter_notebook_config.py
>>>>>>>>>>
c.MappingKernelManager.cull_idle_timeout = 86400
<<<<<<<<<<
(1 day. If idle the kernel is not killed.)

Note that to stop and restart jupyterhub you need also to pkill node after stopping jupyterhub and before restarting jupyterhub, otherwise jupyterhub would find the port busy and don't resume.

For some reasons the jupyter executable is not installed, only one of the various sub-comands, however some programs expect to deal with just "jupyter".
So le't create it:
nano /usr/bin/jupyter
>>>>
#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'jupyter-core==4.6.3','console_scripts','jupyter'
__requires__ = 'jupyter-core==4.6.3'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('jupyter-core==4.6.3', 'console_scripts', 'jupyter')()
    )

<<<<<<

Check available kernels with: jupyter kernelspec list
Reach the notebooks at https://example.org/jupyter 

# ---------------------------------------------------
# 17 - Install Julia

mkdir /usr/lib/julia
cd /usr/lib/julia
wget https://julialang-s3.julialang.org/bin/linux/x64/1.4/julia-1.4.2-linux-x86_64.tar.gz
cd /usr/bin
ln -s /usr/lib/julia/julia-1.4.2/bin/julia julia1.4
ln -s julia1.4 julia

Now, I found that the best way is to install Julia systemwide but havinng each user manage his own packages, including the Jupiter kernels.
What I do is I add a line to adduser.local so that when I add a user with addusers mynewuser a script is run that install a minimal base of Julia packages (including Jupyter kernel) to the user:

nano /usr/local/sbin/adduser.local
>>>
#!/bin/bash

user=$1
su $1 -c "julia /usr/bin/initJuliaRepository.jl"
<<<<
chmod +x /usr/local/sbin/adduser.local

The content of "/usr/bin/initJuliaRepository.jl" is:
>>>
import Pkg

Pkg.update()
Pkg.add("IJulia")
Pkg.add("DataFrames")
Pkg.add("Plots")
Pkg.build("IJulia")
<<<

The relevants part for the apache proxy server are:

<IfModule mod_ssl.c>
	<VirtualHost _default_:443>
	DocumentRoot /var/www/example
        <Proxy *>
           Allow from localhost
        </Proxy>

        RewriteEngine on

        #JupyterHub...
        RewriteCond %{HTTP:Connection} Upgrade [NC]
        RewriteCond %{HTTP:Upgrade} websocket [NC]
        RewriteRule /jupyter/(.*) ws://127.0.0.1:8000/jupyter/$1 [P,L]
        RewriteRule /jupyter/(.*) http://127.0.0.1:8000/jupyter/$1 [P,L]
        #proxy to JupyterHub
        ProxyPass /jupyter/ http://127.0.0.1:8000/jupyter/
        ProxyPassReverse /jupyter/  http://127.0.0.1:8000/jupyter/
	    <Location "/jupyter">
            # Only here as `ProxyPreserveHost On` interferes with Rstudio server login
	        ProxyPreserveHost On
	    </Location>
        [....]
1 Like