Dependency Hell

I’ll start with the problem(s):

(v1.1) pkg> status
    Status `C:\Users\rdboylan\.julia\environments\v1.1\Project.toml`
  [c52e3926] Atom v0.8.7
  [38e38edf] GLM v1.1.2 [`dev\GLM`]
  [54eb57ff] InteractiveCodeSearch v0.3.1
  [e5e0dc1b] Juno v0.7.0
  [295af30f] Revise v2.1.6
  [2913bbd2] StatsBase v0.30.0 [`dev\StatsBase`]

julia> using StatsModels
ERROR: ArgumentError: Package StatsModels not found in current path:
- Run `import Pkg; Pkg.add("StatsModels")` to install the StatsModels package.

Stacktrace:
 [1] require(::Module, ::Symbol) at .\loading.jl:823

That itself is weird since StatsModels is in the general cache at C:\Users\rdboylan.julia\packages. Pressing on …

(v1.1) pkg> dev --local StatsModels
[ Info: Path `C:\Users\rdboylan\.julia\environments\v1.1\dev\StatsModels` exists and looks like the correct package, using existing path
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package StatsModels [3eaba693]:
 StatsModels [3eaba693] log:
 ├─possible versions are: 0.6.0 or uninstalled
 ├─restricted to versions 0.5.0-0.5 by GLM [38e38edf] — no versions left
 │ └─GLM [38e38edf] log:
 │   ├─possible versions are: 1.1.2 or uninstalled
 │   └─GLM [38e38edf] is fixed to version 1.1.2
 └─StatsModels [3eaba693] is fixed to version 0.6.0

After numerous initial attempts to get things working, including getting the latest GLM from github, I concluded the GLM dependencies are just a bit out of date. In particular, project.toml has
StatsModels = "0.5".

I created a local branch and changed that to “0.6”. I did everything I could think of to update dependencies (update; resolve; restarting the editor), but I still get the error shown above. I checked the save file has 0.6 and also noted there is no “other” copy of GLM in .julia\packages.

So what’s going on, and how can I fix it?

Environment

Julia 1.1.1 on MS-Windows 10. Atom/Juno. 64 bit everything.

History

I originally set things up awhile ago (1-3 weeks?) using Julia 1.1.0.
Today I did a series of updates: to Atom and Julia related modules in it; to Julia itself (installed 1.1.1; still have 1.1.0 which I was using); and to the Julia packages. I also installed Revise, including the recommended startup script; i.e., Revise should be active. Told Juno to use the 1.1.1 version (a little surprised that wasn’t automatic). Used external tool to get latest changes for GLM.

Problem doesn’t appear atom-related; same behavior in the vanilla Julia app/terminal.

Have your tried to add StatsModels?
Or alternatively rm GLM , then add StatsModels and then add GLM again?

add StatsModels works because it installs version 0.5 of StatsModels, which is what GLM wants.
dev was failing because head for StatsModels was at 0.6 and GLM had the strict dependency on 0.5

That’s all I need to get on with my work. However, I would still like to understand why my efforts to change the required version of StatsModel in GLM’s project.toml have so far been completely ineffective. Do I need to change project (GLM) uuid or version to persuade julia to look at the rest of the file?

rm GLM might be a little tricky since I have local changes. If it actually deletes the files on disk (not sure if it does) it will remove the changes. If it doesn’t and I try to then dev using the local location, it seems the source and target directories would be the same, which might cause trouble.

IIRC Pkg won’t touch anything that’s been deved. If you’re concerned though, just commit your local changes and push them to GitHub.

This is what git branches are for. dev won’t overwrite if there’s already a folder there, it will just point to it. But if there’s some issue with your code, then doing rm followed by dev won’t change anything.

I’d commit your local changes to a new branch, then checkout master and do git reset --hard. You can also try ] free GLM to see if the latest release is still functional.

I’m a little confused about what’s being asked here, but I think a concept that’s missing from this discussion is that only “top level dependencies” are available with using. The Julia package manager works on the concept of environments. Each package has an environment defined by it’s Project.toml and a Manifest.toml. Regardless of whether you are developing in a package, you are in some Julia package manager environment that has some set of packages directly available through using. These can be listed with ]status. Typically, you also have indirect dependencies, like, for example what StatsModels seems to be for you through GLM. This means that StatsModels exists on your computer somewhere, because it is needed for GLM, but not necessarily that it is available with using. For that, you should add it to your current environment with add or dev, this is intended behavior. GLM will then use the version of StatsModels that you have added, unless it specifies that it needs to use a pinned version.

The reason it works this way is because you would not want using to work on indirect dependencies, because then whether you have valid code would depend on the dependency trees of your dependencies! (Although, you can do, for example using GLM.StatsModels.)

4 Likes

@ExpandingMan: I asked a couple of things, but I think you just answered one: why didn’t using StatsModels work when StatsModels was available in packages? And the answer is that I must explicitly add/dev the package to the current environment. And it is not sufficient for the environment to include a package, such as GLM, that in turn uses StatsModels because that is not a “top-level”, which I read as meaning in the environment, relation.

Thank you for explaining that.

2 Likes

I was thinking rm did what free does. I tried rm GLMand it did not delete the dev\GLM directory with my modified code. Then I used dev to add it back. This did eliminate the changes I made (I had switched branches and changed the dependency, but not committed). I recreated the branch and changed back to StatsModel = “0.6”, but even after restarting the editor the change seems to have no effect.

I committed the changes in git and restarted, but still no joy. I should note my testing method is inverted: since I have StatsModels 0.5, I am waiting for a complaint that GLM requires 0.6–and I’m not getting it.

Maybe dev is working on the master branch, regardless of which branch is active in the filesystem?

This seems odd to me, dev should not alter that directory. It should just see the directory exists and then point to it. Can you reproduce?

No, dev just points to that location. If you have my-branch checked out, and then switch so a totally new project and dev the same package, you’ll still be on my-branch

And now the dev GLM dependency on StatsModels 0.6 has taken effect.
The only thing I can think of that’s changed is that I committed the changes in git.
I did also start a new session, but I tried that earlier and it had no effect.

Well, glad that’s sorted! Guess we’ll just have to let the mystery stand