# Local & remote workflow

**URL:** <https://discourse.julialang.org/t/local-remote-workflow/17780>\
**Category:** Tooling\
**Tags:** question\
**Created:** [November 20, 2018, 3:56pm UTC](https://discourse.julialang.org/t/local-remote-workflow/17780 "2018-11-20T15:56:14Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 20, 2018, 3:56pm UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/1 "2018-11-20T15:56:14Z")

</div>

Suppose I have two machines, LAPTOP and SERVER. LAPTOP is resource-limited but convenient, SERVER is powerful but not super-convenient. Both run Linux, I have standard tools (and I can ask for other ones on SERVER, which is not under my control).

My strategy for working on large problems is to develop a “toy” version on my laptop (eg using small simulated data, a subsample of the data, etc), which I can benchmark and test, until it is ready to run. Then I would like to continue working on the server. I may have to go back and forth a few times.

Some of the code that is packaged gets developed in tandem with the application. Some of the code isn’t packaged, but lives in a git repo. The latter is mostly “scripts”, not modules.

I am curious what workflow people use for this. I can

1. mount the server directory, and work with that, and `rsync` back and forth,
2. sync via private repos on Github or Gitlab (committing the `Manifest.toml`),
3. use Dropbox (but I would prefer to avoid this as I don’t like it much).

FWIW I am using Emacs, so [remote editing](https://www.emacswiki.org/emacs/TrampMode) is a breeze.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [November 20, 2018, 4:35pm UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/2 "2018-11-20T16:35:59Z")

</div>

As an even simpler alternative to 2, you can push branches directly from one machine to another:

```nohighlight
$ git remote add server ssh://me@server.edu/home/me/.julia/dev/MyPkg # do this just once
$ git push server me/fix_issue_13

```

where the last line is pushing a branch of the current repo (MyPkg) to the server for testing.

---

<div class="post-metadata">

**Author:** ![adamslc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adamslc/32/3452_2.png) [@adamslc](https://discourse.julialang.org/u/adamslc)\
**Post date:** [November 20, 2018, 4:36pm UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/3 "2018-11-20T16:36:40Z")

</div>

My approach to distributing packages between the two locations is to create a `git` folder in my home directory on the server, and store the packages there. Then I can create a custom registry, and work with those packages using `Pkg3`.

The drawback to this approach is that there are two copies of each package on the server, but I has sufficient disk space so that isn’t a concern.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [November 20, 2018, 5:00pm UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/4 "2018-11-20T17:00:52Z")

</div>

Definitely good advice here on using Git.  
Another less elegant approach would be to have a shared folder on laptop and server using sshfs  
[https://www.dreamvps.com/tutorials/how-to-use-sshfs-over-ssh/](https://www.dreamvps.com/tutorials/how-to-use-sshfs-over-ssh/)

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [November 20, 2018, 5:04pm UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/5 "2018-11-20T17:04:59Z")

</div>

This Juno workflow works quite well imho:  
[http://docs.junolab.org/latest/man/faq.html#Connecting-to-a-Julia-session-on-a-remote-machine-1](http://docs.junolab.org/latest/man/faq.html#Connecting-to-a-Julia-session-on-a-remote-machine-1)

---

<div class="post-metadata">

**Author:** ![e3c6](https://avatars.discourse-cdn.com/v4/letter/e/e79b87/32.png) [@e3c6](https://discourse.julialang.org/u/e3c6)\
**Post date:** [November 20, 2018, 5:10pm UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/6 "2018-11-20T17:10:25Z")

</div>

Another option which I like is running a jupyter server on the remote and connecting to it from my laptop.

[https://www.blopig.com/blog/2018/03/running-jupyter-notebook-on-a-remote-server-via-ssh/](https://www.blopig.com/blog/2018/03/running-jupyter-notebook-on-a-remote-server-via-ssh/)

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 27, 2018, 10:48am UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/7 "2018-11-27T10:48:49Z")

</div>

Related: for a calculation running for an _ex ante_ unknown amount of time (potentially days), is there a way to somehow notify myself when it is done?

I am currently calling `mailx` to send myself an e-mail, which works, but I wonder what people use.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [November 27, 2018, 1:21pm UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/8 "2018-11-27T13:21:13Z")

</div>

Your cluster manager should have a feature for that. I know that in `slurm`’s sbatch files you can specify an option to that effect.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 27, 2018, 1:22pm UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/9 "2018-11-27T13:22:50Z")

</div>

Thanks, but there is no cluster manager, just a powerful server I can ssh to.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [November 29, 2018, 9:26am UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/10 "2018-11-29T09:26:53Z")

</div>

I discussed workflows for HPC over on the Beowulf list. Someone smart pointed out gitfs  
[https://wiki.archlinux.org/index.php/Gitfs](https://wiki.archlinux.org/index.php/Gitfs)

That sounds very interesting - @Tamas_Papp Could that be a part of your workflow?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 29, 2018, 9:42am UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/11 "2018-11-29T09:42:31Z")

</div>

Thanks, but `gitfs` looks too automated (I want more control), and in any case my git workflow is super-convenient with [magit](https://magit.vc/).

Things are working out smoothly at the moment. In case someone is interested, this is how I do it:

1. set up a private repo on Gitlab, ssh keys and smooth access to the repo on both machines.
2. prepare a pilot run on the laptop on a git branch, push to the repo.
3. pull the branch from the server. if it works out, merge to `master`, otherwise  
a. for minor tweaks, edit remotely via [tramp](https://www.emacswiki.org/emacs/TrampMode),  
b. for larger changes, work on the laptop and push/pull again.

Long runs courtesy of [tmux](https://github.com/tmux/tmux/wiki).

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [November 29, 2018, 10:19am UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/12 "2018-11-29T10:19:18Z")

</div>

Regarding the notifications, I too would use mailx.  
Some Googling reveals this (Python!) utility [About ntfy — ntfy 2.7.0 documentation](https://ntfy.readthedocs.io/en/latest/)  
It uses the Pushover service [https://pushover.net/](https://pushover.net/)

Pushover has a REST API [Pushover: API](https://pushover.net/api)  
So it should be easy to send a notification to your phone and get it to play a little tune when your process ends.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [November 29, 2018, 10:24am UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/13 "2018-11-29T10:24:41Z")

</div>

For a workstation without a cluster manager, I find [Task Spooler](http://vicerveza.homeunix.net/~viric/soft/ts/) `ts` (or `tsp` in Ubuntu etc.) very useful for handling a bunch of processes (e.g., limit number of concurrent processes, run some commands on success of another, automatic stdout storage). Similar tools: `systemd-run --user`, GNU parallel, … what else?

> [@Tamas\_Papp](#):
>
> I am currently calling `mailx` to send myself an e-mail, which works, but I wonder what people use.

I remember using slack notification API via `curl` was not difficult. But that probably is not very different from sending an email.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [November 29, 2018, 3:02pm UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/14 "2018-11-29T15:02:53Z")

</div>

I use unison as I am often swapping between machines, sometimes before I want to commit code  
[https://www.cis.upenn.edu/~bcpierce/unison/](https://www.cis.upenn.edu/~bcpierce/unison/)

I just synch a julia folder with all of my packages and scripts in it. Then edit them on whichever machine I’m on, and run

```julia
unison julia -auto

```

when I need to synch. It connects over ssh, works out which copies are newest or prompts if there is conflict. Then does rsync or similar. I’ve had no dramas after three months using it.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 29, 2018, 3:29pm UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/15 "2018-11-29T15:29:29Z")

</div>

I used to use unison, but stoped about 5 years ago when I learned git. It is a nice tool, but once you are familiar with version control it pales in comparison. No history, no branches, tricky failure modes (eg on divergence).

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [November 29, 2018, 3:38pm UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/16 "2018-11-29T15:38:53Z")

</div>

Sure, I use both! basically both of my machines are identical and many things are synced (not just julia), and git is for pushing to the rest of the world or for when I actually need revisions.

But it doesn’t always make sense to make a revision and push in three or four repositories when I go to sit outside on the laptop or want to try something on the GPU. Even when all my git commands are keyboard shortcuts in rangerfm…

---

<div class="post-metadata">

**Author:** ![cnliao](https://avatars.discourse-cdn.com/v4/letter/c/ce73a5/32.png) [@cnliao](https://discourse.julialang.org/u/cnliao)\
**Post date:** [November 29, 2018, 10:27pm UTC](https://discourse.julialang.org/t/local-remote-workflow/17780/17 "2018-11-29T22:27:50Z")

</div>

I think one key feature that is missing in emacs Julia support is integration of tramp and Julia repl. I tried to follow the example in python.el but that seems specific to comint. If we can figure out how it works with ansiterm then we may get remote executing/completion almost for free. In other editors there’s nothing like tramp and sshfs don’t play well with git.

Currently I just ssh to the remote machine and execute. All editing is via tramp. In case you don’t already know you can alias your server in .ssh/config and setup key based login, so no longish filenames and passwords when editing remote files.