Can someone help me on how to resolve package conflicts with dev'ed packages?

chelate · May 12, 2020, 3:04pm

I’d like to understand better the registry system, and how to troubleshoot this error

ERROR: Unsatisfiable requirements detected for package <<>>

I wanted to use the newest version of SpecialFunctions.jl. The only serious change AFAIK is the addition of spherical bessel functions (why I want to update). However, it seems to be conflicting with a package called ColorVectorSpace, which demands(!) SpecialFunctions v.10. This get’s used by Makie. Also Optim (get’s used by Gadfly). If I want spherical bessels, then I can’t plot! You get the idea.

(@v1.4) pkg> add Gadfly
  Resolving package versions...
ERROR: Unsatisfiable requirements detected for package Optim [429524aa]:
 Optim [429524aa] log:
 ├─possible versions are: [0.15.3, 0.16.0, 0.17.0-0.17.2, 0.18.0-0.18.1, 0.19.0-0.19.7, 0.20.0-0.20.6, 0.21.0] or uninstalled
 ├─restricted by compatibility requirements with ForwardDiff [f6369f11] to versions: [0.20.0-0.20.6, 0.21.0] or uninstalled
 │ └─ForwardDiff [f6369f11] log:
 │   ├─possible versions are: [0.9.0, 0.10.0-0.10.10] or uninstalled
 │   └─restricted by compatibility requirements with SpecialFunctions [276daf66] to versions: uninstalled
 │     └─SpecialFunctions [276daf66] log:
 │       ├─possible versions are: 0.11.0 or uninstalled
 │       └─SpecialFunctions [276daf66] is fixed to version 0.11.0
 ├─restricted by compatibility requirements with KernelDensity [5ab0869b] to versions: [0.15.3, 0.16.0, 0.17.0-0.17.2, 0.18.0-0.18.1, 0.19.0-0.19.7, 0.20.0-0.20.6, 0.21.0], leaving only versions: [0.20.0-0.20.6, 0.21.0]
 │ └─KernelDensity [5ab0869b] log:
 │   ├─possible versions are: 0.5.0-0.5.1 or uninstalled
 │   └─restricted by compatibility requirements with Gadfly [c91e804a] to versions: 0.5.0-0.5.1
 │     └─Gadfly [c91e804a] log:
 │       ├─possible versions are: [0.8.0, 1.0.0-1.0.1, 1.1.0, 1.2.0-1.2.1] or uninstalled
 │       └─restricted to versions * by an explicit requirement, leaving only versions [0.8.0, 1.0.0-1.0.1, 1.1.0, 1.2.0-1.2.1]
 └─restricted by compatibility requirements with NLSolversBase [d41bc354] to versions: uninstalled — no versions left
   └─NLSolversBase [d41bc354] log:
     ├─possible versions are: [7.0.0-7.0.1, 7.0.3-7.0.5, 7.1.0-7.1.2, 7.2.0, 7.3.0-7.3.1, 7.4.0-7.4.1, 7.5.0, 7.6.0-7.6.1] or uninstalled
     └─restricted by compatibility requirements with ForwardDiff [f6369f11] to versions: uninstalled
       └─ForwardDiff [f6369f11] log: see above

etc.
So
1.) how do these packages (Optim and ColorVectorSpace) “know” that they can’t use a higher version of SpecialFunctions?
2.) is there really a dependency issue, and how do I help resolve it if there is and
3.) If there is no REAL dependency issue how do I tell them to sit down and shut up?

xiaodai · May 12, 2020, 3:07pm

They know cos it’s specified in the Project.toml file with version bounds. This is actually a good thing as we dont want an upgrade to break existing functionality.
Just wait for the authors to update their packages to support the latest version of SpecialFunction. Maybe their master branches already do, just check with ]dev Optim etc

Try this:

start a new environment ]activate some_fresh_env
]add SpecialFunctions

Now it should work. Install any other package that you need for the project.

chelate · May 12, 2020, 3:19pm

But is there any way to get v.11 SpecialFunctions to happily coexist with Optim before somebody changes the version bounds without breaking anything? And how does one go about changing the version bounds, and testing if anything is broken on all required packages? Sounds as if it could take a while.

dilumaluthge · May 12, 2020, 3:22pm

You can always suggest to package authors that they install CompatHelper.jl.

(Disclaimer: I am the developer of CompatHelper.jl.)

xiaodai · May 12, 2020, 3:31pm

Well, if u r SURE it’s ok. Just go to the package’s Project.toml and remove the version bounds.

Best to wait for authors to upgrade. I am almost sure they r onto it already

chelate · May 12, 2020, 4:06pm

ColorVectorSpace is managed by @tim.holy and ForwardDiff by @kristoffer.carlsson. If those guys aren’t too busy building the future of scientific computing, maybe they can add a number to their Project.toml files. Will this all be resolved automatically when the registry updates the standard version of SpecialFunctions?

In anycase, I told Pkg I’m a dev! I should be allowed to bend these dependencies by a version number. I used dev, ipso facto I’m a dev, goshdarnit! I have to read the Pkg manual now.

dilumaluthge · May 12, 2020, 4:12pm

You can dev all of the packages, one at a time, and edit their Project files to remove the compat entries.

dilumaluthge · May 12, 2020, 4:13pm

But the real problem is that SpecialFunctions has not made a new release in quite some time. You should open an issue and ask them to make a new release.

chelate · May 12, 2020, 5:32pm

I did this here, but I’m not really sure what I’m asking, hopefully, it will resolve the .toml’s in other packages though.

tim.holy · May 12, 2020, 9:54pm

ColorVectorSpace is managed by @tim.holy and ForwardDiff by @kristoffer.carlsson. If those guys aren’t too busy building the future of scientific computing, maybe they can add a number to their Project.toml files.

Since SpecialFunctions hasn’t actually released version 0.11.0 yet, it hasn’t shown up in CompatHelper. The easier thing to do would be to edit your devved copy of SpecialFunctions/Project.toml back so it claims it’s just 0.10.0.

Marc.Cox · August 26, 2020, 8:56pm

Hi Dilum and Tim @tim.holy

Need either my existing Julia packages running in base Julia v1.04 or v1.05 LTS to run in
base Julia v1.4.2 or vice versa. So I suggested
"… suggest to package authors that they install CompatHelper.jl.”
and the response was
“I used to try to maintain compat with older Julia versions, but noticed that it took
a lot of my bandwidth for little gain and decided to not pursue that anymore.”
as described here >> [ANN] DynamicAxisWarping.jl - #6 by baggepinnen

I like all the cool mathematical and scientific algos Fredrik Bagge Carlson writes,
but I have just as many working in base Julia v1.05 LTS that I really don’t want to break ;
So might you want to reach out to Fredrik and find out what was slowing him down so much
that he gave up on maintaining backward compatibility and address that issue ?

Best !,
-Marc

baggepinnen · August 27, 2020, 4:58am

I can tell you first hand

testing a package takes time, especially since I don’t have anything but the latest version installed locally.
opening the build logs on CI takes time, figuring out what went wrong on an older release takes a lot of time.
fixing things that don’t work on an old version of Julia takes a ton of time. Especially if one of your dependencies has been changed and is not compatible with earlier Julia.

I have around 20 registered packages, and maintaining backwards compatability to outdated versions of julia for all of them is simply not something I want to spend my free time doing.

Tamas_Papp · August 27, 2020, 6:49am

This may not have been the intention, but I find this kind of phrasing somewhat rude. Specifically,

You can reach out to people any time yourself. In this particular case, you already asked the maintainer directly and got an anwser. Pursuing this further is neither necessary nor productive.
Like all people, package devs allocate their own time according to their priorities. When these do not coincide with your preferences, it is not a matter of something “slowing them down” and fixing that.

Marc.Cox · August 27, 2020, 9:18pm

Hi Fredrik @baggepinnen ,

#1 As always Thank You for your really cool mathematical and scientific code,
and also
#2 illuminating the time consuming issues with maintaining backward compatibility;

Best!
Marc

Ps> BTW as a fellow developer (and mathematician) I respect your time invested and the intelligence you must have to accomplish all that you have; and so I apologize if my inquiry ? seemed “rude” somehow ? - if so it was entirely unintentional as I was simply trying to find out if the process kinks could be removed or automatically streamlined somehow by the developers of Compat and CompatHelper.jl etc.

Marc.Cox · August 27, 2020, 9:44pm

Hi Tamas @Tamas_Papp,

Certainly not my intention to be “rude” , also
Re: LTS What are the guarantees of LTS Julia versions? - #5 by saschatimme A long term support (LTS) branch (currently release-1.0 ): an older release branch that will continue to get applicable bug fixes for as long as it continues to be the LTS branch. Extra effort is made to backport bug fixes to this branch

A lot more to be considered about LTS such as risk tolerance of breaking production etc. at that link for Julia’s Release Process but I believe its worth the reading time, and also believe its worth asking questions about too.

Marc.Cox · August 27, 2020, 10:48pm

Hi Dilum @dilumaluthge,Tim @tim.holy, Fredrik @baggepinnen

Just a friendly fyi , I found software that appears to have methods that could automate parts of backward compatibility tests ; This NewPkgEval.jl @@ https://github.com/JuliaComputing/ NewPkgEval.jl automatically obtains multiple versions of Base Julia e.g. v1.05 LTS and v1.4.2 locally for you and provides “ … the following commands to run the tests of a list of packages on a selection of Julia versions : …”

tim.holy · August 27, 2020, 11:49pm

Since this seems to be important to you, perhaps it’s worth clarifying that “long term support” means “things that worked yesterday can still be expected to work tomorrow,” and not “you get access to the latest features.” If you’re on Julia 1.0.x, you need to be prepared for the fact that you may need to use older versions of packages.

To explain why, let me take an example from my own experience, maintaining the ImageMagick.jl package. This package is Julia code but it’s just a wrapper of the libmagick C library. For that, we essentially relied on whatever version the user had installed on her machine. During Julia’s developement prior to 1.3, this led to all sorts of troubles: one test would work on my machine because I had version X.Y.Z of the library, but failed on someone else’s machine because her machine used version A.B.C.

In Julia 1.3 the artifact system was introduced, and this has been an absolute game-changer as far as standardizing the experience. Now I can count on everyone having the same library that I have, and I can replicate their bugs on my own machine. I can’t necessarily fix them, but at least we all get the same experience, and sometimes I can try to write around a bug in libmagick.

So why don’t we backport the artifact system to 1.0? Because it would break the contract: some things might be better, but you would also get some new bugs. That’s a no-no on a LTS—the things that worked yesterday still need to work, even if you also have some stuff that’s broken. It’s just we can’t give you new breakages.

So you have to choose between stability and following the latest development. This isn’t a violation of the 1.x stability guarantee, because that only promises backwards compatibility, not forwards compatibility.

Tamas_Papp · August 28, 2020, 5:37am

I am familiar with the release process, but I wonder if you misunderstand what it implies. Julia (the language) has an LTS version. This does not imply that all packages will keep supporting it — that’s up to the package maintainers.

Also, supporting an LTS version doesn’t have much to do with automated CI. Dropping support for earlier versions of Julia is almost always explicit and intentional, usually done because maintainers need a feature in the new version to make their life easier.

Pinging people here isn’t likely to accomplish anything, but if you are really constrained to the LTS version and want to use new features of a package, you could approach the package maintainers or a third party (eg Julia Computing) and pay them to backport what you need.