Suppose I have two machines, LAPTOP and SERVER. LAPTOP is resource-limited but convenient, SERVER is powerful but not super-convenient. Both run Linux, I have standard tools (and I can ask for other ones on SERVER, which is not under my control).
My strategy for working on large problems is to develop a “toy” version on my laptop (eg using small simulated data, a subsample of the data, etc), which I can benchmark and test, until it is ready to run. Then I would like to continue working on the server. I may have to go back and forth a few times.
Some of the code that is packaged gets developed in tandem with the application. Some of the code isn’t packaged, but lives in a git repo. The latter is mostly “scripts”, not modules.
I am curious what workflow people use for this. I can
mount the server directory, and work with that, and rsync back and forth,
sync via private repos on Github or Gitlab (committing the Manifest.toml),
use Dropbox (but I would prefer to avoid this as I don’t like it much).
My approach to distributing packages between the two locations is to create a git folder in my home directory on the server, and store the packages there. Then I can create a custom registry, and work with those packages using Pkg3.
The drawback to this approach is that there are two copies of each package on the server, but I has sufficient disk space so that isn’t a concern.
Thanks, but gitfs looks too automated (I want more control), and in any case my git workflow is super-convenient with magit.
Things are working out smoothly at the moment. In case someone is interested, this is how I do it:
set up a private repo on Gitlab, ssh keys and smooth access to the repo on both machines.
prepare a pilot run on the laptop on a git branch, push to the repo.
pull the branch from the server. if it works out, merge to master, otherwise
a. for minor tweaks, edit remotely via tramp,
b. for larger changes, work on the laptop and push/pull again.
Pushover has a REST API Pushover: API
So it should be easy to send a notification to your phone and get it to play a little tune when your process ends.
For a workstation without a cluster manager, I find Task Spoolerts (or tsp in Ubuntu etc.) very useful for handling a bunch of processes (e.g., limit number of concurrent processes, run some commands on success of another, automatic stdout storage). Similar tools: systemd-run --user, GNU parallel, … what else?
I remember using slack notification API via curl was not difficult. But that probably is not very different from sending an email.
I just synch a julia folder with all of my packages and scripts in it. Then edit them on whichever machine I’m on, and run
unison julia -auto
when I need to synch. It connects over ssh, works out which copies are newest or prompts if there is conflict. Then does rsync or similar. I’ve had no dramas after three months using it.
I used to use unison, but stoped about 5 years ago when I learned git. It is a nice tool, but once you are familiar with version control it pales in comparison. No history, no branches, tricky failure modes (eg on divergence).
Sure, I use both! basically both of my machines are identical and many things are synced (not just julia), and git is for pushing to the rest of the world or for when I actually need revisions.
But it doesn’t always make sense to make a revision and push in three or four repositories when I go to sit outside on the laptop or want to try something on the GPU. Even when all my git commands are keyboard shortcuts in rangerfm…
I think one key feature that is missing in emacs Julia support is integration of tramp and Julia repl. I tried to follow the example in python.el but that seems specific to comint. If we can figure out how it works with ansiterm then we may get remote executing/completion almost for free. In other editors there’s nothing like tramp and sshfs don’t play well with git.
Currently I just ssh to the remote machine and execute. All editing is via tramp. In case you don’t already know you can alias your server in .ssh/config and setup key based login, so no longish filenames and passwords when editing remote files.