# Handling various types of dependencies

**URL:** <https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850>\
**Category:** General Usage\
**Created:** [August 22, 2019, 2:45pm UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850 "2019-08-22T14:45:38Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [August 22, 2019, 2:45pm UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/1 "2019-08-22T14:45:38Z")

</div>

_(I apologize in advance for the very long post)_

As I use Julia for ever more complex and complete projects, I seem to run into the following problem, where a single “project” contains lots of related code, which will be used in different contexts to perform different _tasks_. (I’m speaking here of a “project” in the Github sense; from here on let’s imagine I’m speaking of a git repository, and reserve the words “package” and “project” to express the Julia concepts defined in the `Pkg.jl` documentation.)

To illustrate what I mean, consider for example a git repository defining a Julia package and containing, in addition to the (re-usable features that the package exports) some code automating various development _tasks_:

1. a test suite, implemented in `test/runtests.jl` (which in turn potentially includes other files) and requiring specific dependencies (in addition to those of the package code itself)
2. possibly some code to post-process the results of test cases (for example submit coverage results to web services such as Codecov or Coveralls). Such scripts typically depend on yet other tools.
3. a `Documenter.jl`-based documentation, in the form of markdown files accompanied by some julia code (in `doc/make.jl`) that actually builds and publishes the documentation. This obviously depends on `Documenter.jl`
4. some benchmarking scripts, which have yet other dependencies (like `BenchmarkTools.jl` or `PkgBenchmark.jl`)
5. possibly some code to post-process benchmarking results and generate plots (like a roofline model, or a plot of run time vs problem size). This code has non-trivial dependencies like `Plots.jl` or `GR.jl`, which you don’t necessarily want to install on the high-performance system where you ran the benchmarks
6. …

The point I am getting at is twofold:

- all this code really belongs to the same git repository; it would not make much sense to have some of it elsewhere. Yet it can be subdivided into several relatively independent _tasks_, which (i) will be used by different types of users, possibly on different types of systems, (ii) are run in different ways, and (iii) require different sets of dependencies.
- we currently have various ways of dealing with these _tasks_. Usually, a top-level Julia program, whose name follows some convention (_e.g._ `test/runtests.jl` or `doc/make.jl`), is used as entry point. The methods (that I know of) to deal with dependencies are variations around the following two themes:
  - dependencies are listed in the `[extra]` and `[target]` sections of the top-level package’s `Project.toml`. To my knowledge, `test` is the only legal target at the moment, but we could imagine having the possibility to list additional (possibly user-defined) targets (`benchmark`, for example). This, in addition to the known path of the entry point, allows tools like `Pkg.test()` to automate the process of running the _task_ in an appropriate environment.
  - the set of source files related to a _task_ is grouped into a sub-folder (like `doc/` for example), which acts as a fully-fledged project, with its own `Project.toml` file defining dependencies. I’ve seen this pattern used not only for documentation building scripts, but also for test-coverage submission scripts, and I’m starting to use it for benchmarking as well. The only (slightly annoying) issue I’m facing with this solution is that one has to explicitly handle environment-related issues: fix `LOAD_PATH` to be able to use the top-level package, call `Pkg.instantiate()` when needed and other such things.

So here are (at last!) my questions:

- are there plans to extend the `[targets]` mechanism for it to be usable for other purposes than testing?
- is there a particular issue with having various sub-project to perform various tasks, each one listing its dependencies in its own `Project.toml` file?
- should we keep these two technique (which solve very similar problems, at least IMHO), or should one be advertized more than the other in the documentation, so that we can perhaps more efficiently build some tooling around it?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 23, 2019, 8:22am UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/2 "2019-08-23T08:22:22Z")

</div>

AFAIU the distinction between `[deps]` and `[extras]` is only relevant when you are working on a package.

If this is a “project”, I would just **put everything in `[deps]`** so that `pkg> instantiate` gets everything — this is otherwise costless if you are not loading a package in a particular script.

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [August 23, 2019, 8:38am UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/3 "2019-08-23T08:38:30Z")

</div>

> [@Tamas\_Papp](#):
>
> AFAIU the distinction between `[deps]` and `[extras]` is only relevant when you are working on a package.

Yes, I’m mostly speaking of packages here. Sorry if that was not clear.

> [@Tamas\_Papp](#):
>
> If this is a “project”, I would just **put everything in `[deps]`** so that `pkg> instantiate` gets everything — this is otherwise costless if you are not loading a package in a particular script.

Even in the case of a simple non-package project, the same problem could happen. Think for example of a research paper (like in this [thread](https://discourse.julialang.org/t/best-practice-modules-and-scripts-for-publishing/25375)), where you might have a program to reproduce the results themselves and store them in data files, and another program to produce the plots shown in the paper.

Depending on the system where you want your program to run (for example on a supercomputer), it might not be convenient to install plotting packages there. The post-processing and plotting stage can then be run on an other system, where graphics dependencies are easier to install.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 23, 2019, 8:46am UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/4 "2019-08-23T08:46:55Z")

</div>

> [@ffevotte](#):
>
> Depending on the system where you want your program to run (for example on a supercomputer), it might not be convenient to install plotting packages there. The post-processing and plotting stage can then be run on an other system, where graphics dependencies are easier to install.

My understanding is that this is what [stacked environments](https://docs.julialang.org/en/v1/manual/code-loading/#Environments-1) are for, but I do not know how to use them in practice.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 23, 2019, 9:10am UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/5 "2019-08-23T09:10:22Z")

</div>

I’m not following Pkg.jl development very closely but given that [Julia 1.2 started supporting `test/Project.toml`](https://julialang.github.io/Pkg.jl/v1/creating-packages/#Test-specific-dependencies-in-Julia-1.2-and-above-1) I guess `[extras]` is going to fade away.

FYI, I also wanted to ease `$subproject/Project.toml` handling so I created a “task runner” package [Run.jl](https://github.com/tkf/Run.jl) ([documentation](https://tkf.github.io/Run.jl/dev/)). The API is that `Run.script("DIRECTORY/SCRIPT.jl")` automatically activate/instantiate `DIRECTORY/Project.toml` then run the script in an isolated environment. There are quick shortcuts `Run.test()` and `Run.docs()` but its useful for general “tasks”; e.g., [running benchmarks in CI](https://discourse.julialang.org/t/pkgbenchmark-jl-workflow/24472/3). All these sub-projects automatically `Pkg.dev`s the parent directory so you can instantiate it even if your main project is not in the registry or does not have URL.

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [August 23, 2019, 9:19am UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/6 "2019-08-23T09:19:58Z")

</div>

> [@Tamas\_Papp](#):
>
> My understanding is that this is what [stacked environments](https://docs.julialang.org/en/v1/manual/code-loading/#Environments-1) are for

Good idea!

Indeed, stacking the “superproject” environment with the environment defined by a “subproject” would probably be ideal.

> [@Tamas\_Papp](#):
>
> I do not know how to use them in practice.

I don’t either. Maybe we lack some tooling and/or documentation in this area.

Here is an attempt fiddling with `LOAD_PATH`. The following tests are run in a directory with the following files:

```julia
.
├── Manifest.toml
├── Project.toml -> deps: LinearAlgebra
└── SubProject
    ├── Manifest.toml
    └── Project.toml -> deps: BenchmarkTools

```

```julia
shell> julia --project --quiet

# Restrict LOAD_PATH so that we don't have access to the home environment
julia> splice!(LOAD_PATH, 1:length(LOAD_PATH)); push!(LOAD_PATH, "@")
1-element Array{String,1}:
 "@"

# BenchmarkTools is not defined as a dependency of the top-level project
julia> using BenchmarkTools
ERROR: ArgumentError: Package BenchmarkTools not found in current path:
- Run `import Pkg; Pkg.add("BenchmarkTools")` to install the BenchmarkTools package.

Stacktrace:
 [1] require(::Module, ::Symbol) at ./loading.jl:823

# However it is a dependency of the SubProject
julia> push!(LOAD_PATH, "SubProject")
2-element Array{String,1}:
 "@"         
 "SubProject"

julia> using BenchmarkTools

julia> 

```

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [August 23, 2019, 9:43am UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/7 "2019-08-23T09:43:30Z")

</div>

> [@tkf](#):
>
> I’m not following Pkg.jl development very closely but given that [Julia 1.2 started supporting `test/Project.toml`](https://julialang.github.io/Pkg.jl/v1/creating-packages/#Test-specific-dependencies-in-Julia-1.2-and-above-1) I guess `[extras]` is going to fade away.

Oh, my bad! I don’t watch change announcements closely enough and had missed this. Thanks!

The documentation speaks of “implictly adding the tested package itself”. Is it a feature specific to tests, or could it be replicated with other subprojects?

> [@tkf](#):
>
> FYI, I also wanted to ease `$subproject/Project.toml` handling so I created a “task runner” package [Run.jl](https://github.com/tkf/Run.jl) ([documentation](https://tkf.github.io/Run.jl/dev/)). The API is that `Run.script("DIRECTORY/SCRIPT.jl")` automatically activate/instantiate `DIRECTORY/Project.toml` then run the script in an isolated environment. There are quick shortcuts `Run.test()` and `Run.docs()` but its useful for general “tasks”; e.g., [running benchmarks in CI](https://discourse.julialang.org/t/pkgbenchmark-jl-workflow/24472/3). All these sub-projects automatically `Pkg.dev` s the parent directory so you can instantiate it even if your main project is not in the registry or does not have URL.

Really nice, thanks! I think `Run.jl` would indeed solve all issues of this type that I encountered.

What I like about Tamas’ idea of stacking environments is that it should allow using the dependencies of several subprojects at once. In other words, if you’re willing to lose isolated environments, you can aggregate a complete environment for all subprojects, as though all dependencies had been declared in the top project.

In this respect, I would say such a solution would combine all advantages: if you want separate environments for testing purposes you can have them; if you want the ease of use of a single environment defining all dependencies, you can have it too.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 23, 2019, 11:53am UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/8 "2019-08-23T11:53:47Z")

</div>

Note

[https://github.com/JuliaLang/Pkg.jl/issues/1251](https://github.com/JuliaLang/Pkg.jl/issues/1251)

[https://github.com/JuliaLang/Pkg.jl/issues/1233](https://github.com/JuliaLang/Pkg.jl/issues/1233)

so I think this is coming.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 23, 2019, 7:58pm UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/9 "2019-08-23T19:58:43Z")

</div>

One of my primary motivations to write Run.jl is reproducibility so [`Run.script` even exclude the default environment (`@v#.#`) by default](https://github.com/tkf/Run.jl/blob/64825fe35b4ed28e302953e77e55840cd396a8e5/src/core.jl#L211-L213). Another reason to avoid stacked environments is that it could break version compatibility. But that’s because CI needs a reliable way to run tasks. For interactive sessions, stacking environments is really great.

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [August 25, 2019, 9:23pm UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/10 "2019-08-25T21:23:07Z")

</div>

> [@tkf](#):
>
> Another reason to avoid stacked environments is that it could break version compatibility

Could you please elaborate a little on this point?

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 25, 2019, 9:47pm UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/11 "2019-08-25T21:47:00Z")

</div>

Consider projects like this:

- Project1

- Project2

I _think_ launching `julia` with `JULIA_LOAD_PATH=Project1:Project2` breaks PkgC because it would import PkgB 1.9 while it needs 2.0.

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [August 25, 2019, 10:01pm UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/12 "2019-08-25T22:01:27Z")

</div>

Thanks, I see why we could not make Project1 and Project2 work together with stacked environments in this case.

Just to be sure I understand you point, what you’re saying is that since both projects can not work together, we should ensure they are run in different Julia sessions. And this is precisely what `Run.jl` automates. Am I right? That makes a lots of sense.

Either way, I think this should not be a problem for me, since the use cases I have in mind involve making sub-projects work together, which are all developed within the same larger project (github project), presumably by the same team.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 25, 2019, 10:10pm UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/13 "2019-08-25T22:10:57Z")

</div>

Yes, I think the hypothetical risk I was talking about is very small especially if you track Manifest.toml in git. You can at least reproduce stacked environments at any moment if there was anything wrong this way.

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [August 25, 2019, 11:05pm UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/14 "2019-08-25T23:05:51Z")

</div>

Sorry - could you clarify this example?

Isn’t the entire point of `Pkg` to look in `PkgA`s Manifest when resolving its dependency on `PkgB`, which can be a different version compared with `PkgC`s dependency on `PkgB` (according to PkgC’s Manifest)?

So there should be no issue loading `PkgA` and `PkgC` at the same time.

Did I misunderstand your comment?

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 25, 2019, 11:14pm UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/15 "2019-08-25T23:14:42Z")

</div>

If both PkgA and PkgC depend on the same PkgB (i.e., UUID is the same), the PkgB loaded in one Julia process is shared across PkgA and PkgC. So, `using PkgB` in PkgC loads PkgB 2.0 even if `Project2/Manifest.toml` has PkgB 1.9.

If PkgB in Project1 and PkgB Project2 are _different_ PkgB which happen to have the same name (i.e., UUIDs are different), then PkgA and PkgC can be safely loadedin one process.

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [August 26, 2019, 12:38am UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/16 "2019-08-26T00:38:48Z")

</div>

Ah, yes. Thank you. I see this now from the updated [code loading](https://docs.julialang.org/en/v1/manual/code-loading/) docs. (Though I don’t understand why this design decision was made.)

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 26, 2019, 2:10am UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/17 "2019-08-26T02:10:43Z")

</div>

> [@hendri54](#):
>
> why this design decision was made

I think you can find an answer in this comparison of Julia’s Pkg to Go’s packaging system:

> [@Pkg3 vs. Go's vgo](https://discourse.julialang.org/t/pkg3-vs-gos-vgo/10154):
>
> The other day I came across the post [A Proposal for Package Versioning in Go](https://blog.golang.org/versioning-proposal) in Go’s blog, and the following passages in particular (edited for clarity) got me wondering about whether these concerns (and corresponding proposed solutions) apply to Pkg3. They start by describing their current approach: […] we all believed the answer would be to follow the package versioning approach exemplified by Rust’s Cargo, with tagged semantic versions, a manifest, a lock file, and a SAT solver to decide w…

Go is treating packages with different major versions as essentially different packages. So the above scenario would have “worked” in the sense that PkgC would load PkgB 2.0 instead of PkgB 1.9.

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [August 26, 2019, 12:22pm UTC](https://discourse.julialang.org/t/handling-various-types-of-dependencies/27850/18 "2019-08-26T12:22:13Z")

</div>

Thanks - a very interesting thread.