Why dependencies of a dependency are not available in the environment?

New to Julia
1

Hi, I have defined an environment (Project.toml) that depends on two (local) packages: package A and B. Package B depends on OrdinaryDiffEq.

Question: Why, when I activate an environment in which I have added (via develop) these two packages, I cannot do using OrdinaryDiffEq?

1 Like
2

The fact that B depends on OrdinaryDiffEq is an internal detail and might possibly change in the future. You can use using B.OrdinaryDiffEq to still access or use Reexport.jl in B to expose it. Maybe even plain export OrdinaryDiffEq.

3 Likes
3

The best answer here is to just add OrdinaryDiffEq yourself. Doing so has no cost, and will make your code clearer.

9 Likes
4

Yes, but then it looks like duplication: now not only the environment but also the package itself declare the same dependencies (there are three of them right now).

5

I totally agree, generally it’s best to add. But I do think there are a few valid usecases for sub-import. For example, A depends on B, but in the documentation of A you want to use B in an example. Now you have 3 options:

  • Add B and with a separate compat entry to your doc env, but this can mess with CompatHelper as it has to bump the version if B in two places simultaneously.
  • Add B with uncapped compat like >0.0.1, because you know it will be constrained by A, feels a bit hacky.
  • Use A.B, which also feels hacky.

Personally, I tend to use 2) but I think this is a valid usecase for 3).

1 Like
6

It’s not though. Direct and indirect dependencies mean different things. As @hexaeder said:

Just to give a concrete example, let’s say I have the following modules in package form:

# Package Foo.jl
module Foo

using Bar

export hello_world 

"""
Returns the `String` "Hello, World!"
"""
function hello_world()
    h = Bar.hello()
    w = Bar.world()
    return string(h, ", ", w, "!")
end

end

# Package Bar.jl
module Bar 

hello() = "Hello"
world() = "World"

end

Now you in your project add Foo, and also really want to use Bar.world() on its own. You can avoid making Bar a direct dependency, and do import Foo.Bar, but if Foo decides to drop the dependency on Bar and simply define hello_world() = "Hello, World!", this would not be a breaking change from the perspective of the Foo, but it would break your package code, and Pkg would not be able to help you prevent that.

Of course, if you’re just developing something for your own use and don’t want to bother with adding the other package to your environment, you don’t have to. It’s your code, you do what you like. But it’s not duplication. It’s free from the perspective of the package manager (the Manifest.toml will be identical), but it has a different meaning and there are different guarantees from the perspective of the broader ecosystem

6 Likes
7

How does it look like duplication exactly? The project or manifest file shouldn’t list any package twice, the minimum statement using A, B, OrdinaryDiffEq doesn’t have redundancies, and it’s routine to repeat module or package names in different import statements.

2 Likes
8

Thank you all for the answers, but I really struggle to understand them :slight_smile:

So, I have a code which has two local Julia packages (in lang_julia folder in the project root), and I want to abstract out the user from this detail, so I want to provide a Project.toml at the project root that declares two local packages as dependencies:

[deps]
PackageA = "07cfd324-e924-40a7-85fa-5e46307e7ad7"
PackageB = "daff4927-7858-4806-96da-915047fb441c"

Now, PackageB depends on OrdinaryDiffEq, so inside the folder lang_julia/PackageB I have another Project.toml:

name = "PackageB"
uuid = "daff4927-7858-4806-96da-915047fb441c"
version = "0.1.0"

[deps]
MsgPack = "99f44e22-a591-53d1-9472-aa23ef4bd671"
OrdinaryDiffEq = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed"
SciMLBase = "0bca4576-84f4-4d90-8ffe-ffa030f20462"

However, I need to include some Julia modules inside the environment, because it is an embedded Julia interpreter, and I need it to load these modules dynamically (it is not known in advance, what module will be requested).

If such a module wants to do OrdinaryDiffEq, then it does not work, as the root-level Project.toml does not include this package as a dependency.

The only working solution that I have so far is to include it in the root-level Project.toml, and then I have duplication, as the same dependency is written in two different Project.tomls.

1 Like
9

Thanks for clarifying, that makes perfect sense. A package being recorded as a dependency across several project files is actually routine. Package managers will attempt to resolve dependencies for a particular environment, in other words it will try to find one version of each package that is compatible with all project files and just load that one for the environment. If it’s impossible or brittle, it’s called dependency hell, and it takes effort across the ecosystem to keep packages compatible with each other across time with some leeway.

As for your case, you should know the versions of OrdinaryDiffEq with which package B is compatible ([compat] section of project file), so you could just use that for your environment. It only appears redundant because you intend to keep the environment and B equally updated for now. But say you’re done with your environment in the next year (locking down its [compat] with upper version bounds) but continue to develop B for the next 10 years; if you have to dust off your environment to reproduce results from a decade ago, you want the environment’s project file to refuse to use the newer incompatible versions of OrdinaryDiffEq and B (which people have pointed out may not even depend on OrdinaryDiffEq anymore). A very common way for old notebooks failing within a few years is a poorly restricted project file (or whatever the language uses for compatibility declarations) or failing to save one at all.

3 Likes