Understanding dependencies of a package

General Usage
1

I am trying to install a package TSAnalysis.jl, which according to its Project.toml only depends on

[deps]
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
Distributed = "8ba89e20-285c-5b6f-9357-94700520ee1b"
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
Logging = "56ddb016-857b-54e1-b83d-db4d58db5568"
StableRNGs = "860ef19b-820b-49d6-a774-d7a799459cd3"
Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

But when adding the project, it seems as it if completely messes up my global environment by downgrading a significant number of packages… see partial output below.

 [1dea7af3] ↓ OrdinaryDiffEq v5.64.1 ⇒ v5.55.1
  [90014a1f] ↓ PDMats v0.11.1 ⇒ v0.10.1
  [65888b18] ↓ ParameterizedFunctions v5.12.1 ⇒ v5.11.0
  [69de0a69] ↑ Parsers v1.1.2 ⇒ v2.1.2
  [f517fe37] - Polyester v0.5.3
  [1d0040c9] ↑ PolyesterWeave v0.1.1 ⇒ v0.1.2
  [d236fae5] - PreallocationTools v0.1.1
  [08abe8d2] ↑ PrettyTables v1.2.2 ⇒ v1.2.3
  [438e738f] ↑ PyCall v1.92.3 ⇒ v1.92.5
  [ce6b1742] ↑ RDatasets v0.7.5 ⇒ v0.7.6
  [fb686558] + RandomExtensions v0.4.3
  [731186ca] ↓ RecursiveArrayTools v2.20.0 ⇒ v2.17.2
  [f2c3362d] ↓ RecursiveFactorization v0.2.4 ⇒ v0.1.13
  [3cdde19b] - SIMDDualNumbers v0.1.0
  [476501e8] ↑ SLEEFPirates v0.6.27 ⇒ v0.6.28
  [0bca4576] ↑ SciMLBase v1.19.2 ⇒ v1.19.4
  [91c51154] ↑ SentinelArrays v1.3.7 ⇒ v1.3.8
  [47a9eef4] ↓ SparseDiffTools v1.16.5 ⇒ v1.13.2
  [276daf66] ↓ SpecialFunctions v1.7.0 ⇒ v0.10.3
  [90137ffa] ↓ StaticArrays v1.2.13 ⇒ v0.12.5
  [2913bbd2] ↑ StatsBase v0.33.10 ⇒ v0.33.12
  [4c63d2b9] ↑ StatsFuns v0.9.12 ⇒ v0.9.13
  [3eaba693] ↑ StatsModels v0.6.27 ⇒ v0.6.28
  [789caeaf] ↑ StochasticDiffEq v6.38.0 ⇒ v6.40.0
  [7792a7ef] - StrideArraysCore v0.2.5
  [69024149] + StringEncodings v0.3.5
  [d1185830] ↓ SymbolicUtils v0.16.0 ⇒ v0.11.0
  [0c5d862f] ↓ Symbolics v3.4.3 ⇒ v0.1.32

For example, how come Symbolics goes from 3.4 to 0.1? Where in the packages DAG is there a dependency to Symbolics? Similarly, OrdinaryDiffEq v5.64.1 ⇒ v5.55.1 which I heavily depend on.

(Yes, I get that I should be creating new envs for my projects, but there are a couple “big” packages in my global env for rapid prototyping)

2

Still, don’t have everything in the main environment. This is exactly why.

2 Likes
3

You should look at the compat, not just which packages it depend on. In particular, this package requires Distributions.jl up to version 0.23, which is from March 2020, that’s likely constraining many other packages:

Well, you know what to do :slightly_smiling_face:

4

So really, is there no way I can have the latest version of DifferentialEquations and TSAnalaysis at the same time? Is the solution here to have staggered envs, so one env has TSAnalysis without DifferentialEquations and a second env has DifferentialEquations and play around with load_path?

5

Open an issue in the TSanalysis repo, maybe it is just about updating the compat entry.

You can also try that, by just changing that in your local dev version and dev the package.

6

What’s the difference in using compat and not just explicitly stating the dependency’s exact version?

7

compat means that any dependency version in that range will be allowed, not that exact version. It allows for more flexibility. Imagine a world where package A depended exactly on version 1.2 of package C, while package B depended on version 1.3 of package C. That would make these kinds of problems impossible.

8

Thanks, so I am now even more confused. Or not so much confused, but a misunderstanding of how the general registry works… see my workflow below.

From the [compat] of the package, we have

[compat]
Distributions = "^0"

which according to semver means anything in [0.0.0, 1.0.0). The latest version of Distributions is v0.25.28. So when I add TSAnalysis, it should just add the latest version right?

SO to test this, I created an empty env and only installed two packages - DifferentialEquations and TSAnalysis. When adding DifferentialEquations, somewhere along the dependency DAG, it installs Distributions … and the correct version (see below)

(testenv) pkg> add DifferentialEquations
    Updating registry at `~/.julia/registries/General`
   Resolving package versions...
   Installed Setfield ──────── v0.8.0
   Installed Symbolics ─────── v3.5.1
   Installed ModelingToolkit ─ v6.7.1
   Installed JuliaFormatter ── v0.18.1
   Installed SymbolicUtils ─── v0.17.0
    Updating `~/testenv/Project.toml`
  [0c46a032] + DifferentialEquations v6.19.0
    Updating `~/testenv/Manifest.toml`
... removed for clarity ...
  [31c24e10] + Distributions v0.25.28.  # <--------- CORRECT VERSION.

Then I add TSAnalysis on top of this, which … again messes up Distributions and downgrades it (and messes up the entire env that Diff Eq relied on as well)

(testenv) pkg> add TSAnalysis
   Resolving package versions...
   # removed for clarity
   Installed Distributions ────── v0.23.8

    Updating `~/testenv/Project.toml`
  [0c46a032] ↓ DifferentialEquations v6.19.0 ⇒ v6.18.0. # WHY??? 
    Updating `~/testenv/Manifest.toml`
  [31c24e10] ↓ Distributions v0.25.28 ⇒ v0.23.8. # WHY???? <<<<<<<<

so why did it downgrade Distributions?

I already know the solution… in the General Registry as posted above by @giordano, the Compact.toml file says

["0.1.1-0"]
Distributions = "0.0.0 - 0.23"

so the ultimate question is then… why is the registry out of sync with the package’s Project.toml?

9

No, what matters is the registry, and that’s what I linked above. You’re confused because you’re looking in the wrong place.

1 Like
10

Thanks, I looked at your link but did not really understand what’s happening in that thread.

A quick and simple solution is to just dev the package since that bypasses the registry compat requirements.

11

With that pull request all packages that did not specify a finite number of breaking versions have been changed to cap the breaking versions allowed and since then we do not allow in the registry packages that do not follow this rule. The package you’re trying to install didn’t release a new version since then (otherwise registration would have failed). They should fix the compat in the package, you can open an issue or do a pull request.

1 Like
12

The way is definitely to open an issue in TSanalysis to update their compatibility.
You can also dev the package yourself, upgrade things on your side, address problems if any, and make a pull request.

In the worst case, you can work with your deved version even if there are no answers on that package repository side.

Side note, when a package is downgraded due to some other package(s), we should indicate which one is responsible. I often struggle to find out which dependency is the one being late.
Maybe an option could be to have a warning when installing a package or one of its dependencies that is not up-to-date regarding compat entries. This would give an easy opportunity for users to open issues regarding compatibility… We could even generate a link in the REPL that if clicked trigger something similar to the compat github action?

1 Like
13

I am going to release a breaking update of TSAnalysis in the following two weeks. This will solve quite a lot of issues and it will include a package renaming. I will also release an additional time series package about in the same timeframe (they should work in tandem). You should be able to do it then.

In the meantime, you can follow the instructions discussed here and use the version on master (locally).