How to install packages which are needed only for development

[ffevotte]

I’m currently preparing a training session, and this process triggers again some questions that I had when initially discovering Julia and which haven’t really disappeared since then.

Here is the current situation: I’m preparing a training session in Julia, where trainees should ideally learn how to develop their own packages. This of course includes managing the dependencies of their own projects/packages (which is a fairly well-documented process). But this also includes using packages such as Revise, BenchmarkTools, Profile+ProfileView… which are not strict dependencies of anything; more like useful elements to incorporate in your workflow. I’d like to provide an easy way for all trainees to install these packages beforehand, so that ideally we will not depend too much on available Internet bandwidth on D day.

Preliminary question:

On my system, I installed all those packages in the default (v1.1) environment. Is that the correct way of doing things? It appears to me that, when a package is activated, you get access to:

1. the package itself
2. all packages that were Pkg.added in the environment of the package
3. all packages that were Pkg.added in the default environment

And Pkg.test checks that actually using the package (and running its test suite) makes use only of the first 2 categories above, as well as Test-specific dependencies added. By the way, it is not clear to me how these test-specific dependencies should be added: is it possible to avoid by-hand editing of Project.toml? Another thing which is not clear to me: in which environment does Pkg.test install test-specific dependencies if they were missing?

I haven’t found any place in the documentation (neither of modules nor of Pkg) where these details are clearly documented. I might very well have missed something; otherwise, I’d be glad to propose PRs to the appropriate documentations.

Back to my current problem: a first option would be to provide a Project.toml file that my trainees would put in their ~/.julia/environments/v1.1/ folder. But then, if some of the trainees have already installed some packages they rely on, my list will override theirs, potentially causing unpleasant surprises later.

Another option (currently my favorite) consists in providing a simple script which goes like this:

using Pkg

# ensure we are in the default (home) environment
Pkg.activate()

# Install dev tools
Pkg.add("Revise")
Pkg.add("BenchmarkTools")
#...

Question 1: Is the above a good way of doing things ?

Question 2: How can I test this? I.e make sure I did not forget any tools in my list. I currently:

1. move the whole ~/.julia directory away;
2. try and install everything in an empty environment, then check that I can perform everything asked in the training session;
3. move my original ~/.julia back.

Is there a better way of doing this?

PS: Sorry for the long question, and thanks to anyone taking the time and effort to read until here…

[fredrikekre]

@ffevotte

On my system, I installed all those packages in the default (v1.1) environment. Is that the correct way of doing things?

It’s one way, I usually use that myself

@ffevotte

It appears to me that, when a package is activated, you get access to:

Yea, well, if those are in the LOAD_PATH (see below).

@ffevotte

is it possible to avoid by-hand editing of Project.toml ?

No.

@ffevotte

nother thing which is not clear to me: in which environment does Pkg.test install test-specific dependencies if they were missing?

It’s a temporary environment that is cleaned up when tests are done.

@ffevotte

I haven’t found any place in the documentation (neither of modules nor of Pkg) where these details are clearly documented.

Stacking of LOAD_PATH is documenter here: Environment Variables · The Julia Language.

@ffevotte

Question 1: Is the above a good way of doing things ?

Sure, seems easiest.

@ffevotte

Question 2: How can I test this? I.e make sure I did not forget any tools in my list. I currently:

1. move the whole ~/.julia directory away;
2. try and install everything in an empty environment, then check that I can perform everything asked in the training session;
3. move my original ~/.julia back.

Is there a better way of doing this?

Instead of moving things you can set DEPOT_PATH to some test-directory, e.g. by setting JULIA_DEPOT_PATH, see Environment Variables · The Julia Language

[ffevotte]

Thank you for the detailed answer!

To recap, IIUC, Pkg.Test runs tests in a context where LOAD_PATH is something like: ["@", "@stdlib"]. I now understand that the current behavior can be explained by piecing together information coming from various places in the documentation. Wouldn’t there be room for a small summarized explanation in module file paths, which would link to other places such as LOAD_PATH (it already does) or Pkg? Most importantly, such a paragraph would tell users what to do when a module they want to use/import is not found, depending on the situation:

• is it the top-level module of their current project? \rightarrow Pkg.activate the project
• is it third-party code that their current project depends on? \rightarrow Pkg.add in the current package
• is it something that they rely on for their workflow? \rightarrow Pkg.add in the home environment
• something else (what?) \rightarrow push! to LOAD_PATH

I’m under the impression that the whole system has been carefully devised in such a way that users almost never have to fiddle with LOAD_PATH, yet this is the first (only?) thing that is mentioned in the modules section of the manual…

As I said, if I understood things correctly (please correct me!) and if there is some agreement to this proposal for a bit more documentation on these matters, I’m willing to prepare a PR.

[kristoffer.carlsson]

@ffevotte

I’m under the impression that the whole system has been carefully devised in such a way that users almost never have to fiddle with LOAD_PATH , yet this is the first (only?) thing that is mentioned in the modules section of the manual…

These docs were written when modifying LOAD_PATH was the standard way of making packages visible to Julia, but with the new package manager this has changed so the docs are indeed a bit “old”.