Dependency problem with FIllArrays

I tagged FillArrays.jl as v0.6 because transpose/adjoint/vec now return an immutable Fill. This is “breaking” because code that assumed these functions are mutable would be broken. But it is very unlikely that any code was actually broken.

Should this have been a patch then?

It sounds like a breaking change so packages that depend on the package should thus not automatically be compatible with this version but instead verify that that they are either not affected by the breaking change, or update their package to be compatible with the new release. When that is done, they release a new version, adding compatibility with the breaking release of the dependency and thus Pkg.update() will not break anyone’s code. I don’t see any other way of doing it.

Assuming that breaking changes do not apply to you and then changing the registry in hindsight when you start getting bug reports about the package not working anymore seems like a much worse way of doing it. Pkg.update() is then very likely to just flat out break users previously working setup of packages.

1 Like

But this has been and still is up to the package developer: we can still do FillArrays = "≥ 0.5.0" and automatically use new versions with breaking changes.

The issue is that users have said they want this behaviour by writing

FillArrays 0.5

in REQUIRE, which has been reinterpreted as FillArrays 0.5 0.6 without the package developers input.

Worrying about users setup breaking is premature before a package is tagged as 1.x. It seems like you are trying to reinvent 0.1.x as the new 1.x.

1 Like

Yes, you can but for now we have a correct start. Upper bounds was too annoying in the REQUIRE format for anyone to use it. Now it is the easiest to use. For the pain this have caused, start going through https://github.com/JuliaLang/METADATA.jl/pulls?utf8=✓&q=is%3Apr+upper+bound+is%3Aclosed+. Accepting stuff like that is very much up to the maintainers of the registry.

How Pkg interprets semver pre 1.0 is documented and is consistent with how Julia itself has been developed and how many other package managers do it (like Cargo).

1 Like

This is just to make sure I understand (I don’t have a strong opinion either way but would like to understand what will happen). In a few packages I have put “relaxed” compatibility requirements (as a practical decision, I accepted that in 0.x versions there would be occasional breakage and planned to be stricter when packages get closer to 1.0 releases). Can I assume that versions were “strictified” according to semver in the registries and will stay strict until I tag a new release, in which case they will respect what I wrote in Project.toml?

As far as I know, that sounds correct.

1 Like

Ok I get it: you are assuming most packages didn’t put upper bounds not intentionally but because it was too hard. Fair enough. I suppose this will sort itself out over time.

1 Like

Can you clarify why you prefer adding bounds manually afterwards instead of having them be correct from the start? e.g. https://github.com/JuliaLang/METADATA.jl/pull/19766, https://github.com/JuliaLang/METADATA.jl/pull/19308, https://github.com/JuliaLang/METADATA.jl/pull/19120, https://github.com/JuliaLang/METADATA.jl/pull/18841, https://github.com/JuliaLang/METADATA.jl/pull/15386, https://github.com/JuliaLang/METADATA.jl/pull/15210, https://github.com/JuliaLang/METADATA.jl/pull/13524, https://github.com/JuliaLang/METADATA.jl/pull/7584 etc. From these it looks like the current behaviour is what you really want? But maybe I am missing something?

3 Likes

Good point, perhaps I’ll start putting upper bounds in. Though it depends on the package: FillArrays.jl is unlikely to change much in between versions.

Upper bounds are “built-in” in the new system in the sense that the next breaking (as defined in the docs) is not allowed. Therefore, something

[compat]
Package = "0.1"

means compatibility with the 0.1-series of releases, but not 0.2 since that is considered breaking.

2 Likes

And “0.1.1” means “0.1.1 <= VERSION < 0.2”?

Yes, see https://julialang.github.io/Pkg.jl/v1/compatibility/

By the way, if you want to encourage this behaviour, perhaps @StefanKarpinski’s gen_project.jl script should be changed as well to not produce ≥ 0.x.y by default?

The script conservatively translates your REQUIRE bounds as literally as possible. Otherwise I’m sure you can see that someone would be complaining that their package which previously installed just fine now won’t install because the generated compat section is much more restrictive than the REQUIRE file was.

It seems like some degree of human judgement about compat bounds is required here which is why I’m not very enthusiastic about automatic translation of the old bounds to the new ones—I think there’s a hard limit on how well it can be done completely automatically and package authors will need to start thinking about and understanding semantic versioning.

1 Like

That argument also applies to the original translation that caused this thread…

Yes, I’m damned if I do and damned if I don’t, which is why I’m mostly ignoring complaints.

4 Likes

Fair enough. Same goes for git knowledge. I see it as an opportunity to learn and improve in something I suck at.

Please don’t be disheartened by complaints. The discussions, coding, and conceptual design devoted to the new Pkg system by you and others is highly appreciated.

4 Likes

@StefanKarpinski I hope you don’t take my comments as complaint. I think it is a great step forward that the new Pkg.jl is raising awareness of package compatibility in the community by making the semver-based constraint [1] the default behavior in [compat] section.

My question is a rather technical one; i.e., if

is true for gen_project.jl and the current state of General. First of all, I’m supposing that I can replace the word “literally” in your sentence with “semantics-preserving” or “faithful.” I don’t see it how it connects with the next sentence if not.

Isn’t this what is happening here? And isn’t it because Package = "x.y.z" is not semantically equivalent to Package x.y.z? Why it is done this way even though [compat] can represent it as Package = ">= x.y.z"? Or maybe my assumption is wrong because the dependency resolver is changed?

But you also said

So maybe it’s rather a “social” or “pedagogical” issue rather than a technical one? That is to say, the decision was made to give a narrower range of compatibility so that package authors are forced to think about the compatibility issue seriously?

[1] Side note: I’m not sure talking about compatibility in sub-1.0 software is strictly semver compliant. At lease I think the use of Package = ">= 0.y" does not have to be so frowned upon if your package is sub-1.0 as well.

1 Like

I don’t know if it’s damned if do or don’t, but definitely if you do and don’t, especially in a random way. So far we have:

  1. Upper bounds added when moved from METADATA to General
  2. No upper bounds when generating Project.toml from REQUIRE
  3. Upper bounds ignored by Registrator.jl (https://github.com/JuliaLang/Pkg.jl/issues/1177#event-2304129269)

I thought I understood what to do (use MyPackage = "0.3" in a tagged release to only support 0.3.x and retag when MyPackage.jl v0.4 comes out) but now I have no idea what we are supposed to be doin…