My understanding is that exported functions are always external but not all external functions are exported necessarily. So you can’t know what is external just looking at the code. You have to look at the documentation to figure out what the package has made “external”.
2 Likes
You brought up at (I didn’t want to answer there/derail the main topic?):
It could be argued those packages would depend on 1.x+1 but that wouldn’t solve the problem, because then yes those versions would no longer be installable in that Julia, and would force use of older versions of the packages, that also wouldn’t work.
So it got me thinking, should it be the other way around, Julia having [compat] on some packages? I’m just not aware that Julia has a Package.toml file per se (at least not the compiler), only some stdlibs of it.
These are very much special cases, and I believe core devs may be involved anyway, and this would require a new version tagged to work with a (an otherwise) breaking Julia master, that’s about to go stable.
In theory any package can depend on internals of Julia and break this way. And I’m not proposing a long list of all those. But your argument is that there are only few (plus its indirect many dependants). Would this be a good solution, would automatically fix the dependants, and most of the ecosystem?
That would not make very much sense since you are presumably running the incompatible Julia version, which cannot be changed then and there. What you are really talking about is retroactively changing the compatability of a package to a new Julia version. This can be accomplished via a pull request to the General registry, if needed:
1 Like
I wasn’t aware possible/allowed, but still I’m not sure it helps. Not just if simply people haven’t retroactively done that, but also if you have a package, lets say Cassette, that works in 1.9.3 and then you update to 1.10, and it no longer works. Presumably you had internet (or diskettes…) to get you the new Julia, but for whatever reason when you first run that Julia you might not have, e.g. in an air-gapped environment.
I could see that you would then want to force people to check the registry on first use (and have a dilemma), and if the package people only start updating the registry after release of new Julia then there is no future point where it’s no longer unsafe for sure to have a cached copy of the registry.
I think you mean in this scenario the new 1.10, and it would have its [compat] for a newer version of Cassette that you do. You can still run Julia, with all compatible packages, but not incompatible such as old Cassette (or its dependants). But then you know. You will get an error for old incompatible such packages “you need to download a new version”. Julia doesn’t strictly need to bundle those new versions, but potentially some of those, could be (upgradable) stdlibs.
The registry contains exactly the compatability information between a package and the Julia version. If you’re on an airgapped version, you probably should just obtain the registry tarball and use that.
Here’s an example with for TypedSyntax.jl retroactively requiring Julia 1.6 for all versions:
Older versions of TypedSyntax.jl do not have a Julia 1.6 entry in Project.toml:
ERROR: Unsatisfiable requirements detected for package TypedSyntax [d265eb64]:
TypedSyntax [d265eb64] log:
├─possible versions are: [1.0.0-1.0.12, 1.1.0-1.1.11, 1.2.0-1.2.2] or uninstalled
├─restricted to versions * by an explicit requirement, leaving only versions [1.0.0-1.0.12, 1.1.0-1.1.11, 1.2.0-1.2.2]
└─restricted by julia compatibility requirements to versions: uninstalled — no versions left
As a small reminder: Linux has managed to not break the user space for over 30 years.
True. However if you had watched the discussions about systemd … it was controversial.
1 Like
systemd is not part of the kernel
well, some systemd maintainers found it a good idea to patch the kernel (actually udev) to “support” systemd development (because init is more important than kernel, and yes, that’s sarcastic).
linux kernel has a “don’t break userland” strategy for a reason. You will see similar discussions upthread (i have not read all) about julia, the language, julia the implementation and julia, the ecosystem of basic packages. It’s reported (in all good faith) that julia never breaks with minor releases - except in some cases.
It’s hard and close to impossible to not break userland. imho linux manages this beause other unixes didn’t do before and created very strange situations.
They learned from others.
I am not quite sure what you mean by that. The Linux ABI changes all the time: new features are added, and old ones are obsoleted then removed. Backward compatibility is only guaranteed for 2 years.
Which is as it should be. All nontrivial software projects need breaking changes to the API once in a while. The important thing is that transitions are managed smoothly and with minimal disruption.
The stability of the exposed interface is a distinct question. In an ideal world everything would be bug-free, but of course bugs and regressions happen, both in commercial and open source projects. Again, the important thing is that they are fixed in a timely manner.
1 Like
Applications are supposed to continue working.
Linus Torvalds is well known to be VERY strict about that, and finds strong language to contrast his point.
1 Like
Unintentionally breaking the API without warning is – as that email makes clear – a nonstarter, but as far as I can tell, Linux kernel API methods can absolutely be deprecated and removed with due process and warning. The current kernel major version is 6.x.x which certainly indicates that there have been breaking changes in the last 40 years.
I am wrong. Thanks @Palli and @ShalokShalom !
1 Like
You assume this solely because of the version number?
Linus has also been on record, that version numbers don’t mean much to him, and he increases the major version number with no clear intent or message.
I also remind that he is quite displeased about ensuring backwards compatibility in the sense of 20-year-old software running fine on Linux, and then the distributions destroying all of that due to how they package.
And I remind that even people who are pretty hard on ‘no-API-breaks’ themselves, do consider Linus Torvalds to be a bit fanatic about that. 
I assume its all about priorities. Not breaking certainly has drawbacks, and for a language like Julia argubly less so than for a Linux kernel. And still, its certainly possible. The question is only, if we want to pay the price.
And we dont need to pretend, we could not provide absolute API stability, its fine to say that we find it sometimes worth it to change the API. I certainly think, it is so the case, sometimes.
And I agree that a proper announcement and a decent amount of time (6 months and up) does seem to provide a sensible fundament to change.
2 Likes
Are the stdlib packages also tested as a part of a PkgEval run? Or are updates to stdlib packages provided more of a leeway to introduce breaking changes in minor Julia releases?
Linux ABI and API is much more stable than you think, I would actually like to know of a single intentional breakage to syscalls, or an accidental one that hasn’t been fixed.
For a compiled program the ABI is important, and you note “only guaranteed for 2 years”, I read
“backward compatibility for them will be guaranteed for at least 2 years. Most interfaces (like syscalls) are expected to never change and always be available.”
This never changing ABI/API is like the Julia API (also ABI?) guarantee. But Julia has internals and they change. Linux of course also has internals, but you can’t access them. It’s just not technically possible to read (or write) kernel space (or other process memory).
No software need API changes to the kernel (nor really changes for any other API). It does mean duplicated APIs, some old API cruft.
You don’t make an API vs ABI distinction. The ABI is important if you do a syscall directly (and I suppose the promise for 2 years, is so they can drop obsolete archs like possibly Motorola 68000, it may apply to /proc, I’m not sure, not regular user apps). Most likely your program doesn’t do much of it, you don’t need very often, you talk to the (dynamically linked) Libc library, and it’s a system library occasionally updated. Even if the ABI of the kernel were to change, then only the Libc might need to change and your program would never see the ABI (or any API) change.
This is the ideal, what very few programming projects live up to (in or out of Linux; the userspace, i.e. syscall, interface is stable, but true the kernel API, for drivers, intentionally isn’t, long story on pros and cons). To be fair, if you don’t change your Julia app (e.g. Manifest) nor the Julia runtime then you will also expect your program to run correctly forever.
APIs may be deprecated I guess, in a way that they are still kept around forever (see Python’s soft deprecations, likely similar). I don’t know of a single Linux API dropped,because Linux doesn’t follow semver. Semver is about allowing you to break, and tell you when, Linus doesn’t want that to ever happen to Linux. Possibly neither should Julia.
EDIT: Here I see one “ABI break” that seemingly though isn’t (if you use the “old” 32-bit time_t, but then your program will break in another way unless you’re ok with modular time…):
Y2038: An example of an ABI break
The Linux ABI is best understood by considering the example of the ongoing, slow-motion “Y2038” ABI break. In January 2038, 32-bit time counters will roll over to all zeroes, just like the odometer of an older vehicle. January 2038 sounds far away, but assuredly many IoT devices sold in 2022 will still be operational. […]The Linux kernel has already moved to a 64-bit time_t opaque data type internally to represent later timepoints. The implication is that system calls like time() have already changed their function signature on 64-bit systems. The arduousness of these efforts is on ready display in kernel headers like time_types.h, which includes new and “_old” versions of data structures.
[…]What precisely is in the Linux stable ABI anyway?
Understanding the stable ABI is a bit subtle. Consider that, while most of sysfs is stable ABI, the debug interfaces are guaranteed to be unstable since they expose kernel internals to userspace. In general, Linus Torvalds has pronounced that by “don’t break userspace,” he means to protect ordinary users who “just want it to work” rather than system programmers and kernel engineers, who should be able to read the kernel documentation and source code to figure out what has changed between releases. The distinction is illustrated in the figure below.[…]
Ordinary users are unlikely to interact with unstable parts of the Linux ABI, but system programmers may do so inadvertently. All of sysfs (/sys) and procfs (/proc) are guaranteed stable except for /sys/kernel/debug.[…]
But what about other binary interfaces that are userspace-visible, including miscellaneous ABI bits like device files in /dev, the kernel log file […] Naturally, “it depends.”
[…]
Conclusion
The kernel ABI stability guarantee applies to procfs, sysfs, and the system call interface, with important exceptions. […] Newer features like BPF programs present as-yet-unanswered questions about where exactly the ABI-stability border lies.
Even if the Linux kernel API/ABI is stable, programs depend on more, Lib etc. of a full Linux distro, and they may not be as strict:
2 Likes
Oh, this makes a lot more sense to me now.
1 Like
I mean not possible to directly read or write the memory (of the kernel) through (non-API). If it applies, then you can, through an API, or debug “API”/well “file”, like mentioned in my edit /sys/kernel/debug.
If you think, then: why doesn’t Julia do the same, prevent access to internals? For Julia (also applies to any other programming language), then Julia’s internals are in the same address space as the rest of the program. It’s not possible to 100% prevent access (that requires an MMU, or running in a different process). You could still prevent like with “private” as in C++ and other languages, which doesn’t fully stop you from accessing or changing. There are pros and cos to disallowing that (see though possibility of public being added as a Julia keyword), it’s considered ok to not prevent, just make clear you shouldn’t access internals (usually, C++ has friend for exceptions, some packages like Cassette could be considered friends of Julia, i.e. Julia’s internals…), and only very few packages doing that.
2 Likes
Yeah, I simply assumed that there is some difference, and you confirmed that there is one. I did not know, at all, how that would look like.
Thanks for clearing that up. You happen to be very knowledgeable and helpful.
1 Like