How do I update an environment in Julia?

[danielsoutar]

Hi, a colleague of mine and I were arguing about environments in Julia in a project we have.

Let’s say we have a standard project:

$ ls
Project.toml ... Manifest.toml ... src test ...

I might have some version restrictions in the Project.toml, so I might amend them. For example:

- SomePackage = "^0.7.1"
+ SomePackage = "=0.8.0"

What would I need to do to ensure the environment reflects this change? Does it need to be a clean build? And what constitutes a clean build?

My understanding (obviously correct me if I’m wrong) is that you would do this in Pkg. You might instantiate in the current environment. Alternatively a ‘clean build’ would mean deleting the Manifest and regenerating it via activate .; instantiate in Pkg. The latter could be important for ensuring that the update to SomePackage propagates to all other packages.

My colleague says that in order to have a clean build, you need to remove the entire .julia directory in the home directory. This seems extreme to me, but maybe this is occasionally necessary.

I am somewhat conflicted because I suspect there may be cases where doing each of these makes sense. Can someone explain with reference to other factors such as:

• Whether a particular package might install an external C library or require one
• Multiple versions of the same package
• Package version requirements
• A ‘library’ versus an ‘app’ (i.e. the latter is not a precompiled package/library but rather a script that acts as a consumer of Julia packages)
• Any combination of the above

Any clarification greatly appreciated!

[ffevotte]

@danielsoutar

My understanding (obviously correct me if I’m wrong) is that you would do this in Pkg. You might instantiate in the current environment.

Yes, this is correct. Pkg should be used to manipulate environments, and instantiate is the command you want to create (“instantiate”) an environment that is consistent with the current contents of Project.toml. The description of the complete environment is saved in the Manifest.toml.

EDIT: To go a bit more into the details: Pkg.resolve is used to build a consistent environment and save it into Manifest.toml, and then Pkg.instantiate is responsible for taking the environment described in Manifest.toml and making sure all listed packages and versions are installed on your machine.

If no Manifest exists, calling instantiate will implicitly call resolve first. If a Manifest already exists but is not consistent with the current description of the project, you’ll need to re-resolve and instantiate will tell you so.

@danielsoutar

Alternatively a ‘clean build’ would mean deleting the Manifest and regenerating it via activate .; instantiate in Pkg.

Do you mean “clean” in that this simulates what would happen to a new user of the package? Then that is true.

If my understanding of Pkg is correct, this should be equivalent to running update instead of resolve. IIUC, the difference between these two commands is that resolve tries to determine a minimal set of changes (w.r.t. the current environment described in Manifest.toml if it exists) that are needed to make the environment consistent with the requirements described in Project.toml. On the other hand, update tries to find an environment that is as up-to-date as possible (i.e. it will install newer versions of dependencies when available, even if it is not strictly necessary to conform to the project requirements).

@danielsoutar

The latter could be important for ensuring that the update to SomePackage propagates to all other packages.

Not sure what you mean by that. Running resolve will ensure that any update you make in the compatibility bounds of your project will be reflected in the environment. Running update will additionally try to bring into the project any newer version of your dependencies that might have been released since the last time you instantiated/updated your environment.

@danielsoutar

My colleague says that in order to have a clean build, you need to remove the entire .julia directory in the home directory. This seems extreme to me, but maybe this is occasionally necessary.

Please do not delete the entire .julia directory. It is usually not needed and may have unintended consequences (including the loss of some data that can’t be recovered from the Internet).

See for example this recent discussion about deleting the .julia directory:

Danger of deleting .julia?

From thread Please refrain from doing such dangerous suggestions again. What is the danger of deleting .julia?

[mkitti]

1. Consider using the Pkg REPL or the Pkg API to do this rather than manually changing the Project.toml.

julia> using Pkg

help?> Pkg.pin
  Pkg.pin(pkg::Union{String, Vector{String}}; io::IO=stderr)
  Pkg.pin(pkgs::Union{PackageSpec, Vector{PackageSpec}}; io::IO=stderr)

  Pin a package to the current version (or the one given in the PackageSpec) or to a certain git revision. A pinned package is never updated.

  Examples
  ≡≡≡≡≡≡≡≡≡≡

  Pkg.pin("Example")
  Pkg.pin(name="Example", version="0.3.1")

(jl_Yna2si) pkg> ?pin
  pin pkg[=uuid] ...
  pin [--all]

  Pin packages to given versions, or the current version if no version is specified. A pinned package has its version fixed and will not be upgraded or downgraded. A pinned package has the symbol ⚲
  next to its version in the status list.

  Examples

  pkg> pin Example
  pkg> pin Example@0.5.0
  pkg> pin Example=7876af07-990d-54b4-ab0e-23690620f79a@0.5.0
  pkg> pin --all

2. If you modify this project and there are other projects which depend on it, you should also consider bumping the minor version number.