I often run into a problem when installing packages with a large number of dependencies where my environment will be forced to downgrade to old versions because a single dependency hasn’t updated its version requirements.
For example, when installing Turing.jl, my default environment downgraded the following packages:
Is there any way Julia could manage packages more like Rust’s cargo does? Cargo is able to install different versions of packages in parallel for each dependency requirement.
how does that work? I mean either they work or they don’t right? In Julia because you can potentially using [all your packages], they all have to work, thus the downgrading. If you know you don’t need them to work together, just use separate environment (which is recommended anyway regardless of whether or not the issue you mentioned is actively annoying you)
Say you want to use both packages A and B in the same environment. They both depend on package C, but package B requires an old version of C.
Julia forces one to downgrade C for both packages A and B. But it could instead install two versions of C and keep track of which version to use for A and which version to use for B. I’m probably oversimplifying, but I believe that’s what happens with Rust.
As a side note, this error message isn’t very helpful. It’s not clear to which packages the version ranges are referring:
(@v1.8) pkg> add Turing@0.18
Resolving package versions...
ERROR: Unsatisfiable requirements detected for package Libtask [6f1fad26]:
Libtask [6f1fad26] log:
├─possible versions are: 0.1.1-0.5.3 or uninstalled
├─restricted by compatibility requirements with Turing [fce5fe82] to versions: [0.4.0-0.4.2, 0.5.3]
│ └─Turing [fce5fe82] log:
│ ├─possible versions are: 0.5.0-0.18.0 or uninstalled
│ └─restricted to versions 0.18 by an explicit requirement, leaving only versions 0.18.0
├─restricted by compatibility requirements with AdvancedPS [576499cb] to versions: 0.5.3
│ └─AdvancedPS [576499cb] log:
│ ├─possible versions are: 0.1.0-0.2.4 or uninstalled
│ └─restricted by compatibility requirements with Turing [fce5fe82] to versions: 0.2.4
│ └─Turing [fce5fe82] log: see above
└─restricted by compatibility requirements with Libtask_jll [3ae2931a] to versions: 0.1.1-0.4.2 or uninstalled — no versions left
└─Libtask_jll [3ae2931a] log:
├─possible versions are: 0.3.0-0.5.1 or uninstalled
└─restricted by julia compatibility requirements to versions: [0.3.0-0.3.2, 0.5.0-0.5.1] or uninstalled
A common recommendations is to not use you default environment (except for Dev tools like Pluto, IJulia,. Cthulhu, BenchmarkTools etc).
And instead create an environment for each thing you are doing, with only the packages you need.
It won’t completely solve your problem, but it will vastly decrease it’s impact and frequency.
Out of curiosity, how many different environments do you use?
I try to keep it to under 10, but that’s not nearly enough. Just for plotting I’d probably need half a dozen different environments. In practice though, I use only about 5 and I destroy and reinstall each of these once or twice a month.
Yes, Rust installs two versions - julia doesn’t allow that, since only one version of a package can be loaded at any given time. When using per-project environments this usually isn’t that much of a problem, but I can see how for larger projects this may be undesirable. Due to the dynamic nature of julia, loading more than one version would (hypothetically) lead to a lot of recompiled code, since you can’t assume that compiled code for version A1 of Pkg A would also work for a dependency requiring version A2. Since julia doesn’t (yet?) ship precompiled binaries or shared objects of julia code like Rust could (and since Rust can check a lot more statically than julia), there’s not much to be done here for now in terms of reducing code size/compile effort.
You should also be aware that while rust allows this, it can lead to problems as mentioned in their docs, with similar explanations and problems as in julia and is best avoided. It’s a philosophical difference that julia doesn’t deal with this problem like rust.
And there also was another post (which I can’t seem to find right now) where the conclusion was basically to fix the version bounds of the third package, since minor versions should be backwards compatible anyway (though not for 0.x releases, which should be an incentive for packages to release a proper major version 1.0 to make that part of dependency resolution work).
–
And since I accidentally posted in the wrong thread and discourse doesn’t just let me move posts, here is some extra text to fool discourse into thinking I wrote something different.
I am a very particular different user profile.
I rarely really “use” Julia.
I maintain many many packages.
Each of which is an environment.
And I often use it as such doing dev --local to try changing packages together.
I have a folder called where I keep a clone of each package I work on.
With like 100 or so package environments
Which is the vast vast majority of my use of Julia.
Other than that five or six.
Much less used.
Default, one for testing out various mixes of packages in the AD system (this gets recreated a lot).
One for Invenia’s main production system (sometimes 2), one for our hyperopt system (sometimes 2), one (sometimes 2) for running experiments and analysing results.
But I rarely use these, because mostly I work on packages.
But environments are cheap, every JLSO file is an environment with everything needed to load that file.
Every Pluto notebook is by default an environment.
I create 2 or 3 environments with activate --temp every day to try something or reproduce a bug.
I’m not sure what you mean. If you have 20 projects, use 20 different environments. Whenever you start a new project, just to ] activate . . You have 5 main environments and switch between them depending on the session? Just make new projects.
Just to add to what Peter and Lyndon said the other key (I would actually say the most important) reason to have project specific dependencies is reproducibility - run some analysis, finish it, file it away. Have to return to it two years later? Not a problem, your project specific Manifest.toml will ensure that everything just runs.
I see, thanks for the detailed answer. I guess my reluctance to keep a large number of environments came from the challenge of remembering which environment to invoke. Do you automate this? like maybe putting something like the following into a function?
dir = expanduser("~/my-package-v99/")
push!(LOAD_PATH, dir)
import Pkg
Pkg.activate(dir)
Base.active_project()
using my-package-v99
yes sure, I do keep one environment per project and typically have 5 or 6 projects active at a given time. But as I work on a project I add/remove packages, which tends to mess things up by causing unwanted (and unexpected) downgrades. Do you create an environment on the fly every time you experiment with a new package? Thanks.
Yes. I get the point that waiting for package operations and precompilation is a pain. One thing that can help is Pkg.offline(), which will make it so that Pkg tries to use package you already have installed first.
But it would be nice for you to activate a new project by cloning an existing Project.toml, potential taking advantage of any pre-compilation. But then when you modify that environment, it doesn’t change the original toml file. I don’t think something like that exists.
I’ve confused myself more than once when having multiple projects opened in VSCode at once, with the result that I would Pkg.add in the wrong environments by mistake. My miserable workaround is I use the Julia IDE for experiments and temporary environments, IntelliJ for when I pull up some older project to remind myself of how I did things there, Juno for my side project and VSCode for my main project. It’s my mental map at this time. Pretty weird, but having multiple projects opened in adjacent tabs inside VSCode hasn’t worked too well for me.
)
