Forward compatibility and stability of Julia vs. Packages

General Usage
21

Python has no concept of a private API either, but only a widespread convention of obfurscating private methods and members by adding leading _.

Correct me if I’m wrong, but I haven’t heard anyone complaining (at least on this forum) that Python packages are always breaking for them and that’s a reason they come to Julia.
Assuming that is correct, would it be too much to ask the Julia community besides annotating mutating methods with !, to also annotate private methods and members with a _?

1 Like
22

The design of private-API designators is really a separate issue RFC: marking private and public APIs · JuliaLang/julia · Discussion #48819 · GitHub

1 Like
23

Sure, that’s potentially also an interesting test, and not only for x-1 but all the way to Julia 1.0 — it’s the earliest version current Julia promises compatibility with.

But still, this kind of test relies on package authors actively updating their packages. Technically, it doesn’t tell anything about julia backwards compatibility! The only honest test in this regard would be to restrict all code to that written at the time of the previous (or earlier) Julia release — no updates.

If many packages use Julia internals (and seems like this is the case!), it doesn’t mean that Julia doesn’t follow semver, of course. But it does mean that in practice old code often breaks, and it would be nice to make users aware of that.

1 Like
24

Sure, if old versions of SortAlgorithms and StatsBase will work on the actual 1.9 release, I’ll take those words back. As I understood, though, this is considered fine, and no fixes are planned.
I don’t write down these things, but pretty sure there were similar (maybe, less wide) breakages earlier as well. Popular packages release updates following Julia releases of course, but it’s practical breakage nevertheless.

25

I’d certainly be interested in hearing of the results if you decide to run the tests you’re advocating! Given that nightly is widely tested in the package ecosystem, I don’t think I can rule out the possibility that you are making an important point. But until we see some data, I think (hope?) you’re overestimating the amount of breakage.

2 Likes
26

And after test failures, package authors tend to fix them (if real and not nightly transient).
I may try running such tests on a few packages at some point though. But pretty sure that many packages that worked on eg Julia 1.2 don’t work on 1.8 without changes.
Unfortunately, running package tests can be too brittle compared to actual usage. Tests may check for specific exception types for example, or show formats — while this doesn’t practically affect users.

27

But pretty sure that many packages that worked on eg Julia 1.2 don’t work on 1.8 without changes.

I’d agree from personal experience that’s true, but until proven otherwise I’ll assert with high confidence that it’s more often a mistake in the package ecosystem (issues with [compat] bounds) than breakage from Julia.

Unfortunately, running package tests can be too brittle compared to actual usage. Tests may check for specific exception types for example, or show formats — while this doesn’t practically affect users.

Agreed with the reservations, but something is better than nothing. When we’re developing new Julia versions, if it passes on 1.x-1 but fails on what will become 1.x, we actually dig into why and generally hold the release until that’s fixed. Obviously this takes a ton of effort, but since we go to that effort for new releases, I think it’s incumbent on you to do the same if you’re going to convince us that we have a backwards compatibility issue.

Packages that fail their tests on 1.x-1 don’t tend to trigger the same scrutiny, for obvious reasons (we can’t fix the entire package ecosystem).

2 Likes
28

What exactly do you mean by “mistake with compat”? If package works on Julia 1.x but doesn’t on 1.y, it’s definitely breakage by Julia. It may or (likely) may not be semver violation, but this doesn’t matter for the user: he cannot update Julia and keep the same package versions.

It’s often fixed by updating the package code and releasing a new version. So, this is a Julia minor version breaking perfectly working code.

Whether this is an issue or not is subjective. But some code and packages routinely stops working after Julia updates nevertheless.

1 Like
29

A couple of examples. I don’t mean to pick on anyone here, these are examples of errors that I’ve seen (and made myself) again and again:

It’s often fixed by updating the package code and releasing a new version. So, this is a Julia minor version breaking perfectly working code.

Occasionally. More often we fix Julia. And sometimes we discover bugs that have been lurking for years: make sure value wrapped in mutable is not garbage collected by timholy · Pull Request #682 · JuliaGraphics/Gtk.jl · GitHub fixed a rare GC-related issue that very rarely caused segfaults on older versions of Julia but reliably caused segfaults on 1.9. Yes, there are sometimes where we “migrate” package code, but we ourselves submit the fixes and it’s quite rare that the fixes were needed for a package that was targeting Julia’s public interface.

But some code and packages routinely stops working after Julia updates nevertheless.

Without proof, this is becoming a pointless waste of time.

1 Like
30

This is the “proof” that minor Julia versions break existing package code. What other confirmation is needed?

31

Actual examples. I can’t think of a case where the package was targeting the public interface; I’m not saying it has never happened, but I can’t think of a single case. In contrast, I can think of lots of cases where packages broke each other. Heck, a couple of us fixed one in just the last 24 hours: remove the hack added in #498 to avoid `InitError` by aviatesk · Pull Request #500 · aviatesk/JET.jl · GitHub.

So again, you’re making a strong claim. Prove it with examples that you’ve checked actually trace back to breakages of Julia’s backwards-compatibility guarantee.

3 Likes
32

May I repeat again: I don’t claim that Julia doesn’t respect semver. Aside from few rare exceptions, Julia only changes behavior in what’s considered “internals”, so everything is correct wrt semver and “official” guarantees. But many packages depend on these internals, and so updating Julia breaks them.
“Officially” everything is fine, but in practice old packages and code stops working.

33

If a package breaks because it relies on internals (julia or other package), that’s totally fair game. I’m struggling to understand what the goal with complaining about such breakage is - what is the actionable item someone can do that helps with this? Other than “provide an API for querying what API is” and using that to communicate what the API surface is, there really isn’t anything julia itself can do, and individual packages breaking really can only be fixed on a case by case basis (other than not relying on internals in the first place).

I do agree that packages relying on internals is bad, especially if (as is subjectively often the case) the internal access could very well be upstreamed. I really do hope that we can get some version of the discussion @jar1 linked above sooner rather than later, because with every passing version the backlog of stuff that needs to be inspected for whether it’s API grows and grows. That is not sustainable, and I’d rather have people complain about getting warnings for using internals than them complaining about breakage.

3 Likes
34

This thread started with my minor observation that updating Julia tends to break at least some (even popular) packages, that’s it. Unfortunately, there’s little what can be done to fix that indeed.

35

Ok, but that’s a social/cultural problem of package authors using internals (which they shouldn’t - even (and especially) when they themselves wrote those internals). As a user of such packages, please do complain to package authors about them using internals. “Move fast and break things” is a curse on otherwise quality software, and taking the time to do things right is, while sometimes annoying in the short term, VERY healthy for the longterm trajectory of the ecosystem.

6 Likes
36

How to find it out in advance, before these internals actually break?

37

It started by you doubting the claim:

Julia is much more stable than the package ecosystem.

I think (?) we’ve agreed that the claim is actually true.

38

Sorry, maybe it was better for me to also quote the next sentence:

And this is often not the case in (my) practice. Many packages are user-facing with little or no dependents, so updating them has little potential to break anything — at least there’s no large cumulative effect as with updating Julia.

39

Other than running CI on pre-release versions or release candidates, you don’t. That’s exactly why using internals is bad and why fixes of the sort of if VERSION <= foo and lathering on more internals, likely to break at the next release again, should be complained about loudly.

2 Likes
40

SemVer isn’t good enough. I sympathize with your view, for the average user, if something breaks for any (undocumented) reason at all (with an upgrade of Julia, or even any package), then it’s a problem (the solution, for now, not upgrade anything, including Julia, keep the Manifest), and the ideal would be it CAN’T happen (encapsulation). If you couldn’t rely on internals of Julia (or some package), then things would be better. I’m not sure e.g. you can do that with Java. I understand the JVM has excellent compatibility with new versions. [Yes, old versions may have security issues, so you need to upgrade.] I don’t have enough practical knowledge about e.g. Python (or Perl), to know it it’s as good, or better than Julia that way. [Python is similar to Julia, in that you can access (and mutate) internals (of structs), no “private”, so a potential (actual?) problem there. It might extend to the standard library of Python, but I do not know it it extends to the core of Python, its VM. One solution for Julia and Pyton I see is to aggressively trim the standard library, then less stuff can actually break, and fewer internals to access, but you would just move the breakage to the package ecosystem… so I’m still concerned with the semantic issue.]

Julia is the first language I follow the closely the development of (even first open source project I really follow), and it does seem to have a fair amount of regressions, before release, I guess that’s normal. Sometimes regressions persist after release (or new bugs), why we have minor updates. I think the only sane option, for now, is to not upgrade (for production) to new releases immediately, e.g. 1.9. But the older version, and I expect for 1.8 too, is immediately dropped from support as soon as new are out. So you’re out of support unless you use LTS.

That’s one perspective (ok by SemVer). I would want to know what those packages are. For e.g. C++, with encapsulation, that wouldn’t be possible (because of private by default, well “friend” is an intentional exception), and you would get a compile time error. In Julia only at best a run-time error, I guess that’s likely, but you can’t check for sure (I suppose the “hating problem” theoretically, though in many cases practically possible with static analysis). Are there any tools that check for use of internals? Could packages that do such be marked somehow? And such be made “friends” of other packages, in the C++ sense?

Is that what happens? Do you mean with --depwarn=yes?

People will use internals because they can. I’m conflicted about if Julia should allow it, but it would now be a breaking change (ironic) to disallow such. Would it be possible to begin with to have a runtime flag to show such?

I guess the good thing is that, even when packages break, the ecosystem does react quickly. Or it does, have you had to wait long until fixed (for some specific package, how long)?