Although “production-ready” is a vaguely defined term, PackageCompiler has a 2.x version number and it does not mention anywhere in its docs itself that it should be considered alpha/beta quality (nor any warning to that effect). So it is absolutely not unreasonable to consider it ready for general use, aka “production-ready”.
6 Likes
Semver concerns API stability, not production-readiness or some measure of quality. There’s an element of truth in that proprietary software usually releases as v1, but 1) PackageCompiler isn’t a product, and 2) this isn’t a hard rule, even Semver only strongly suggests it. In practice, people can and often decide on stable API before having the staff, time, tests, and documentation to justify any confidence in production use. The absence of upfront warnings about regressions or quality in the documentation doesn’t promise quality at all; it could just as easily suggest authors don’t have the time.
Vague definition aside, I think it’s a reasonable expectation for production-ready software to lock down the versions of dependencies known to work, but there evidently is the conscious decision to let PackageCompiler be used for newer Julia versions and the resulting regressions be patched over time by volunteers. We don’t need to even skim the open issues, the release notes already show that intention. Something to consider is that PackageCompiler is inherently sensitive to Julia’s internals, not a library that can comfortably rely on v1 API. It seems pretty clear to me that PackageCompiler is an independent, usable experiment until Julia itself can subsume its purpose, which I gather is juliac’s intention.
1 Like
not to get all “yes it is” “no it isn’t” but I daresay I’ve absolutely seen this feature actively promoted. maybe not necessarily by the PackageCompiler.jl developers and rather mostly by excited early adopters, but I can see how those groups can be quite hard to distinguish for those without a deep familiarity with the Julia community
6 Likes
I’m not contesting that, and that’s a far cry from an official recommendation or any sort of promise from the core Julia developers. I’d hazard a guess that LoopVectorization, still a v0 package, was promoted even more by Julia users, and that was almost deprecated to @inbounds @fastmath for Julia v1.11+ when the developer shifted the vectorization work. It ended up getting funding for further maintainance to undo the regression of many dependent packages after significant outcry, that’s how successful its promotion was.
1 Like
To simplify the discussion, maybe let’s just assume that PackageCompiler is “stable” as indicated by its >1.0 version number, not just in terms of its API, but also in terms of being “not experimental”. It’s reasonable for people to assume that they can rely on its documented functionality. That does not mean that it has no bugs, or that the existence of an unfixed bug would prevent a new release.
And it especially does not mean that Julia should not make releases because PackageCompiler (which is not even a stdlib package) has some bug, or that the Julia project needs to communicate about this bug in any way.
I agree this particular bug is pretty severe, but at least it’s a known bug with a clear workaround. And I do think that the existence of a GitHub issue is sufficient to track and communicate this. I’ve never seen release notes of any projects mention new, unfixed bugs.
1 Like
Hoping a mod can split this subthread into its own topic about indicators of reliability e.g. production-readiness.
I agree that a dependency (Julia here) isn’t at all responsible for and shouldn’t be held back by a dependent (PackageCompiler here), but it’s an entirely reasonable expectation that reliable, let alone production-ready, software’s releases be delayed by regressions that seriously compromise performance or usability. I don’t think people would put up with it even for perpetual betas. If PackageCompiler was intended to be reliable, I’d expect a patch to prevent updates to Julia v1.11 until the regressions are fixed, if not preemptively restricted to validated Julia versions.
Fair enough. I might make the distinction that this is a new bug, and is a “regression” only in the sense it was then identified as having worked in earlier releases. So I’m giving them the benefit of the doubt that the cause for this bug might still just be hard to track down or hard to fix. I would indeed raise an eyebrow if a project had a regression in the sense that something broke in their existing test suite, and they decided to make a release anyway. But, maybe the PackageCompiler developer should have treated this particular issue with more priority. I just don’t really want to point fingers or make too many demands on what level of reliability a particular project should guarantee. I would consider PackageCompiler a relatively important package, so maybe they should strive for a higher level of reliability. But this is is definitely a side-discussion of a side discussion, and probably should be split off 
I’m definitely not blaming anyone for feeling frustration after being bitten by this bug.
1 Like
Could you check if make sure 1 thread is used in julia processes by KristofferC · Pull Request #1041 · JuliaLang/PackageCompiler.jl · GitHub fixes it?
4 Likes
So what does that mean for some of the most popular and most used Julia packages with a 1.x or higher version number then? Or for (perceived) maturity and quality of Julia packages in general? For example, based on number of stars on juliapackages.com there’s Plots.jl (currently v1.40.12), ModelingToolkit.jl (currently v9.72.0) or Symbolics.jl (currently v6.38.0)? All are libraries and/or frameworks, so according to your reasoning above the actual version number has no relation to package maturity, actual value in daily use, as the version only says something about their API only. Are you really trying to say these packages are not production-ready, or should not be expected to be moderately bug-free, or even usable to some extent at all?
If so, then that’s a very narrow interpretation of semantic versioning, as I’m sure the vast majority of package users haven’t read the semver specification to fully grasp all the details, and also don’t really care. I’d say the de-facto interpretation of version numbers is anything <= 0.x is to be considered not fully stable and not fully usable yet, v1.x clearly signaling a first stable and functioning release, and anything >=1.x signals continued iteration and further maturing.
The other thing that wise library/package/program/application developers do is manage expectation in their documentation. If you don’t clearly mention something is experimental, “still in development”, immature or anything to that effect then users will expect something that is at least usable and valuable to some extent.
3 Likes
Yeah, technically @Benny might be correct that “stable” and “production-ready” are separate concerns. “Stable” in that interpretation just means “makes semantic-versioning guarantees to track breaking/non-breaking changes”. But as the SemVer specification itself says in it’s FAQ:
If your software is being used in production, it should probably already be 1.0.0
Quite honestly, I don’t even know what it would mean for a package to be stable but not-production-ready. Maybe, “not enough test coverage to be relied on”. But then, if there are no tests, how can you possibly make SemVer promises: How would you detect a breaking change, if not via tests?
So this goes “slightly” beyond SemVer in the narrowest sense, but I think it’s more than reasonable for people to assume that >1.0 indicates not just “stable”, but also “production-ready”. I’d think that >90% of people would make that assumption, and that @Benny is just being a bit idiosyncratic here.
I would certainly strongly urge people to not tag their packages as 1.0 if the package is not production-ready. If I saw a package registration for a version v1.0 that didn’t have any tests, I’d leave a blocking comment along the lines of “are you sure this is really ready for 1.0”.
P.S.: If we want to go down too much further this rabbit hole of discussion, @mbauman should probably also split this off into a separate thread about what SemVer means in relation to “stable” and “production-ready”.
2 Likes
I didn’t mention any of those packages, so no, it suffices to say I didn’t make any of the claims you imagined.
In actuality, you’re insisting that semantic versioning has more guarantees than API stability, which is a strictly narrower interpretation than mine. Nothing I said denied the possibility that a v1+ package is mature, reliable, production-ready, etc. It’s simply also possible that it’s not the case, so you have to look at more factors beyond a few numbers, a few of which you named.
“Production-ready” would be a bit high for me, but yes I agree there should be a strong basis for reliability because formally stabilizing the API is a high cost that shouldn’t be paid prematurely. It’s very possible that testing exposes flaws in the API, and once you need to making breaking changes, you’re snowballing into greater numbers. But I don’t assume that there’s never a reason for it, and evidently people have their own opinions about reliability and going 1.0. It’s clear that PackageCompiler is more reliable than its previous iteration, even if it falls short of production-readiness.
1 Like
Many thanks. However, it will be a working workaround as hotfix and these question remains:
- why can Julia 1.10 work with several threads and Julia 1.11 not? This way Julia 1.10 is for me 40 % faster in compiling the official example lib of PackageCompiler
- There no note/info/explanation about this regression. Loosing so much compilation speed when upgrading from 1.10 to 1.11 is not negligible and users should be informed, for example by adding info to the release notes.
- why does PackageCompiler not output any warning or error if the number of threads don’t fit (for Julia 1.11 if JULIA_NUM_THREADS > 1 and for Julia 1.10 if e.g. JULIA_NUM_THREADS is set to 16 but there are only 12 CPU cores)
- in your PR there is no comment that others can later understand why the execution was restricted to only one thread, e.g. that this is a temporary workaround.
2 Likes
You made general claims about semantic versioning and API stability, so those would apply to all packages?
No, I don’t care about semantic versioning specifically, as I suspect many software packages don’t really follow the actual specification, some just loosely, some none at all. My point was about the way version numbers (and changes of versions from one to the next) are interpreted informally.
1 Like
Ok, does that PR work for you though?
4 Likes
I can’t rule out something in your unknown code, but IIRC, Julia v1.10 is what parallelized some parts of sysimage building, custom ones being what PackageCompiler is currently mainly responsible for. 1.11 made quite a few changes to internals, so it’s possible that an inadvertent interaction compromised custom sysimages, which is still undiscovered according to the relevant Github issues. Note that it was already possible in v1.10 for memory to run out and stall the system during multithreaded building of large sysimages. In general, multithreaded memory allocations multiply the burden on the GC, and the line between a program’s OutofMemoryError or a bona fide memory leak isn’t obvious.
In my opinion, there are 2 ideal ways this goes for independent development:
-
prioritizing reliability: Dependencies are strictly locked down to versions known or expected to work by
[compat]. If what was expected to be an acceptable update significantly regresses or outright breaks a feature, a patch quickly further restricts the dependency. Careful work and releases take the time to support expected breaking changes and work around regressions. -
prioritizing availability: Dependencies are mostly lower bounds for support and automatic upper bounds for major versions. Users risk running into issues that tests didn’t cover, and they are encouraged to report issues or contribute fixes, tests, warnings, etc. In one very conspicuous place, other users, especially new ones, are informed of the discovered caveats of usage and practical workarounds if any.
As strongly as I disagreed with people here on (1) being the only acceptable approach or how SemVer is used and interpreted, I do agree that transparency is beneficial. I can only speculate that the much smaller development team and occasional contributors, both volunteers AFAIK, makes even this much difficult, let alone the sort of development work in (1). Things can be better for users, but the necessary conditions don’t exist on demand alone.
2 Likes
I do know that SemVer isn’t a strongly enforced standard, let alone the only one for versioning. Informal interpretations would have even less of a consensus, not more. Again, nothing I said implied your preferred interpretation of SemVer to be non-compliant with SemVer, in fact it’d be quite nice if it were mandated in the standard.
“Semver concerns API stability, not production-readiness” does apply to all those packages. Again, it does not imply “these packages are not production-ready…or even usable…” To spell it out, the set of packages with stable APIs contains reliable or production-ready packages. Nothing in the SemVer standard or this thread denied so; you were arguing against a position of your own invention. Respectfully, that’s the last comment I’ll contribute to this particular subthread, whether you accept it is up to you.
FWIW, I’ve tried to find a second meaningful topic-split here several times but couldn’t come up with separate topics that made sense independently of each other. Let’s just bring the pedantic semantic versioning vs. production-ready nitpicking subthread here to a close. Evaluating library quality is a fairly universally challenge, and the meaningful indicators (and their relative weights) can vary between dev and language communities, sub-groups, and individuals. If anyone wants to continue that discussion — and you’re very welcome to — please open a new dedicated topic that starts with a clean slate.
10 Likes
Is this really true? I can imagine 1.11 being slower in itself, but I don’t think anything in the PackageCompiler process should use threads (except a part of the very last step in outputting the sysimage and I am not sure that even uses the number of julia threads). If you turn off threading, does 1.10 also become 40% slower?
Could you explain what part of PackageCompiler you think gets faster with your threads environment variable?
I just tested myself and didn’t notice a difference (on 1.10):
# Without threads
julia> using PackageCompiler
julia> ENV["JULIA_NUM_THREADS"]
"1"
julia> Threads.nthreads()
1
julia> @time create_app("examples/MyApp", "MyAppCompiled"; force=true)
...
150.910366 seconds (1.92 M allocations: 151.919 MiB, 0.01% gc time, 0.28% compilation time)
# With threads
julia> using PackageCompiler
julia> ENV["JULIA_NUM_THREADS"]
"10"
julia> Threads.nthreads()
10
julia> @time create_app("examples/MyApp", "MyAppCompiled"; force=true)
...
148.749071 seconds (3.82 M allocations: 291.721 MiB, 0.03% gc time, 0.28% compilation time)
5 Likes
PackageCompiler gets stuck with multiple threads for some cases (e.g. `create_app` never finishes on Julia v1.11.1 when `JULIA_NUM_THREADS` is set · Issue #990 · JuliaLang/PackageCompiler.jl · GitHub and PackageCompiler never ends on julia 1.11.0 with --threads=6 · Issue #963 · JuliaLang/PackageCompiler.jl · GitHub), so using a single thread becomes the workaround for those. Don’t know about the performance impact in general, though.
Okay but that was the whole question.
2 Likes