Sync'ing Julia packages between multiple machines

General Usage
1

Suppose you use multiple machines and personally manage your Julia environment. What do you do to sync packages between those machines?

I’m not a “developer” and I don’t need serious version control at all. I’m happy with always upgrading to the latest versions of the packages.

The simplest method I think of now is to write the output of Pkg.status() to a text file on one machine, send it to another machine (or share it via cloud drive), and read and parse it on the other machine to install the missing packages.

If all my machines were of the same architecture I could just rsync ~/.julia/ but would that work between Intel mac, M2 mac, and Linux?

2 Likes
2

I suppose the problem here is that you do not have one Project, but multiple projects, so just keeping the Manifest.toml synced is not enough?

1 Like
3

Thanks for your response, but

I suppose the problem here is that you do not have one Project, but multiple projects, so just keeping the Manifest.toml synced is not enough?

I know very little about “Project” and I know nothing about Manifest.toml.

With this in mind, can you elaborate on your advice?

I’ve just searched for Manifest.toml under ~/.julia/ and found a lot of them. Do you mean I sync all of them and do nothing more than that to sync packages?

4

The best approach in my opinion is to use a project specific environment which would have its own Project.toml file for compatibility bounds and optionally a shared Manifest.toml which would provide constraints on all dependencies (and dependencies of dependencies etc.). If you use the global version of those files in /.julia, you are likely to have divergences.

You can make project specific environments with the package management system or with PkgTemplates.jl (which is similar but for a full package). Using a private Github repo would be one way to synchronize across machines. I know you said you are not a developer, but those tools are good for your stated purpose and are not too complicated to learn with a bit of practice.

2 Likes
5

To be more specific:

In one machine, you start Julia and do:

mkdir MyProject
cd MyProject
julia # start Julia

In Julia:

julia> activate . # activate the MyProject (current dir) project

julia> import Pkg; Pkg.add(["Plots", "DifferentialEquations"]) # your packages

This will create two files in the directory MyProject: Project.toml and Manifest.toml. The first contains the list of packages of you added to your project, the later the exact version of every package installed in your environment. These two files is what you need.

When running your code/scripts, you do:

cd MyProject
julia --project script.jl 
#or
julia --project # to start a interactive julia section with the project activated

Then, copy those two files (or the complete directory) to your other computer. The first time you will then do:

cd MyProject
julia
# in Julia
julia> import Pkg; Pkg.instantiate()

This will install all the necessary packages with the exact versions listed in the Manifest.toml file.

You can then run your script exactly as in the original machine.

(as mentioned above, this directory can be associated to a github repository, in which case you can clone the repository in other machines, instead of “copying” the data, the main advantage being the version control, backup, easy of use for other people).

4 Likes
6

Project.toml is a file that stores the package names of installed packages for a given environment and Manifest.toml stores the package names and also exact versions of packages. A pair of Project.toml/Manifest.toml uniquely and completely define an entire package environment and can be used to recreate it anywhere else.

If you’re working mainly in the global environment, these files are in somewhere like ~/.julia/environments/v1.9/. So copy just those two from one machine to the other at the same location. Then on the new machine, open up Julia and run pkg> instantiate. Now all packages at the exact same versions are installed on the new machine.

3 Likes
7

Thank you all for your kind help! I now understand (better) how a Project is managed. Because I’m so far happy with just one default “environment”, I’d probably go with

~/.julia/environments/v1.9/ . So copy just those two from one machine to the other at the same location. Then on the new machine, open up Julia and run pkg> instantiate . Now all packages at the exact same versions are installed on the new machine.

This is exactly a kind of method which I had suspected would already exist before I posted my initial question (and before starting to write my own script that would sync the packages using the Pkg package).

I’ll try to sync these files via github because I agree with

Using a private Github repo would be one way to synchronize across machines. I know you said you are not a developer, but those tools are good for your stated purpose and are not too complicated to learn with a bit of practice.