Should I use `]add` or `]develop` to access local packages from my main environment?

General Usage
1

Scenario: I have a package, say Widgets, which exists in a public Github repository (but not in the Julia package repo) at github.com/maxkapur/Widgets.

I have also git cloned Widgets to my home folder for development purposes.

Normally, if I want to play in the REPL with one of the functions in Widgets, I open a terminal in ~/Widgets and run julia --project.

Now, however, I would like to use the Widgets package in my “main” Julia environment. That is, the one I get when I run julia from a terminal without the project flag. (What is the proper name for this environment?)

In other words, I want to add Widgets to my main Julia project.

From looking at the docs, it seems like there are six ways to do this: run julia, then enter at the REPL

  • add ~/Widgets
  • add github.com/maxkapur/Widgets
  • develop ~/Widgets
  • develop github.com/maxkapur/Widgets
  • develop --local ~/Widgets
  • develop --local github.com/maxkapur/Widgets

Which one should I use? I am interested in the following issues:

  • If I update the package on Github, will my local environment automatically update?
  • If I update the package on Github, will I be able to update my local environment using ]update?
  • If I update the package on Github and then pull changes to ~/Widgets, will my local environment automatically update?
  • If I update the package on Github and then pull changes to ~/Widgets, will I be able to update my local environment using ]update?

(I am typing this on my phone in a subway; please pardon any syntactical errors.)

2

You definitely want one of

The --local flag is only relevant if you’re using a GitHub link, as it determines where the clone will be saved (--local uses a relative path whereas --shared uses an absolute path). It’s irrelevant if you’re specifying a local path directly.

The difference between using a local path and a GitHub link is that in the latter case, a clone will be fetched and used, whereas in the former case your local copy will be used. In no case will an update on GitHub be automatically fetched, unless you specifically pull the changes, or update your project. In each case, what you’ll be using is ultimately the local copy of the project, and not what’s on GitHub.

  • If I update the package on Github, will my local environment automatically update?
  • If I update the package on Github, will I be able to update my local environment using ]update?
    issues:
  • If I update the package on Github and then pull changes to ~/Widgets, will my local environment automatically update?
  • If I update the package on Github and then pull changes to ~/Widgets, will I be able to update my local environment using ]update?

The answer to the first is no. If you have added your package using develop --local link, the answer to the next three is yes. The pulled changes will be automatically used, and you don’t need to update your project every time.

Even if you’re directly editing your local clone that the project is pointing to without pushing to GitHub, the changes will be incorporated automatically.

If you have a local copy at ~/Widgets which is synced with the copy on GitHub, and you add your package as develop --local link, you’ll end up with two copies of the project, and your project will use the local copy that is at cwd()/dev. This may be out of sync with the copy at ~/Widgets, unless you keep fetching changes from GitHub. This is probably not what you want. You want to use

develop ~/Widgets

and edit your package as you want. The local changes to the package will be incorporated in your project. You may push your changes to GitHub, but this is not necessary for you to receive the updates, as they’re available locally.

You probably don’t want add, as this will track specific commits or tags in the project, and won’t incorporate uncommitted changes. In case this is what you want, you will need to explicitly use ]update to change the commit that is being tracked.

1 Like
3

Thank you!

What happens, then, if I modify ~/Widgets/Project.toml in a way that breaks my dependency tree? For example, suppose I am on v0.23 of a package Fidgets.jl in my main environment and I edit the deps in ~/Widgets/Project.toml to require Fidgets.jl v0.24? Will Julia sort everything out if I do ]update?

After I do this, does calling julia --project from ~/Widgets become superfluous?

4

Will Julia sort everything out if I do ]update ?

yes, it should, as I understand it.

After I do this, does calling julia --project from ~/Widgets become superfluous?

There are two projects here: one being your main Julia environment, and the other being your package Widgets. If you are planning to edit the contents of your package, you need to use julia --project from ~/Widgets to access its dependencies. On the other hand, if you just want to use Widgets in your main environment, you don’t need to activate it; in fact, you shouldn’t, as it might lead to issues in resolving dependencies. In the second case, you should simply run julia followed by using Widgets, once you have run develop ~/Widgets in your main environment.

5

Follow-up question: suppose I decide to do something silly and develop my own local package called Plots.jl (even though a package of the same name already exists in the repos). Is there a way I can ]develop my project under an alias, say ]develop ~/Plots as SillyPlots, that won’t break other packages in my environment that have the real Plots.jl as a dependency?

Or what about this: Suppose I haven’t installed Plots.jl yet, nor any package that names Plots.jl as a dependency. I create my own Plots.jl package locally that does something completely unrelated to plotting and ]develop ~/Plots. Now what happens if I ]add StatsPlots, which depends on the real Plots.jl? Will it try calling plotting functions from my local project and error, or will it install the real Plots.jl as an alias?

For that matter, what happens if I try to ]add Plots in this scenario?

I am tempted to try these things myself but I don’t want to break my environment…

6

Aha. So suppose I have Fidgets.jl v0.24 installed on my main environment, and Widgets.jl names Fidgets.jl v0.23 as a dependency. If I now call ]develop ~/Widgets, then my main environment will downgrade me to Fidgets.jl v0.23, right?

7

If you have strict compat settings in Widgets that pins Fidgets to v0.23, and this downgrade is possible in the main environment (i.e. no other package in the main environment depends on Fidgets v0.24 specifically), then yes, it should downgrade Fidgets to v0.23.

8

For playing around with packages and to test these things you can open a temporary environment with

] activate --temp

Since packages are disambiguated via an UUID you can have different packages with the same name as indirect dependencies, see 10. Project.toml and Manifest.toml · Pkg.jl

1 Like
9

That’s an easily solved problem. Create an empty directory and point the environment variable JULIA_DEPOT_PATH there. Now you have a pristine Julia installation and can play around all you want without affecting anything you have in your .julia.

1 Like