About Julia's development policy regarding Windows

[uwestoehr]

I hoped to get a discussion about the issues of Julia’s development but all I got is telling me that I do the things wrong, Julia’s development is fins as it is.

If Julia wants to be attractive for more users, it must please the users, and not tell them what they should do different. Julia must try to offer what users need. If they need to compile their Julia module as an app, then you must provide this.

The argumentation here always drifts away from the topic of the thread - the development policy. And you see by yourself, if new features of a release don’t work, there is a QA issue. Also if the release notes don’t mention regressions and remedies.

I was constructive, made a list wit proposals. But there is no discussion about the proposals.

@goerz

The release notes say that a --trim flag was added. It doesn’t say anything about the scope of that feature or how stable it is.

Such a statement is why I opened this thread. How can it be that a new feature is announced but one cannot expect it working?
This violates any QA I know. As I said, QA can be handled differently, but there must be somehow a checklist, a set of rules, protocols, whatever.

@goerz

Maybe take some time to actually get familiar with the community and Julia’s development process before making these kinds of demands on “QA”.

I searched for that but I cannot find a publicly available checklist how releases are made, no QA policy, no policy how PRs are reviewed. And why am i not allowed to ask for QA when i am not happy with the quality? Things can only be improved if one agrees that there is a problem.

---

@goerz

You missed the top of that page. The official installer is juliaup.

I go to the Downloads page and there are several options. There is nothing written that I have to use juliaup. There is no info that the manual installation is expected to have issues.
To get juliaup, I first must to install it using the Windows store. And only because I have Windows, does not mean I also want to become a customer of the Windows store. We bought out Windows licenses from our computer supplier. We only have a contract with them, not with Microsoft.

@goerz

Nonsense. The CI for Julia runs on all supported platforms.

So this bug is nonsense? This is the issue I am facing.

@giordano

Did you notice the floating pencil-shaped button in the bottom right corner of the page?

Yes. Clicking it leads me to a GitHub page. As I wrote, that would mean I have to make a PR to change the docs. This is a high hurdle and not friendly to new or average users.

[goerz]

@uwestoehr

So this bug is nonsense?

THIS BUG IS NOT IN JULIA. And, it’s an open issue. Hence, there is no test for its resolution. That’s what QA would look like: after someone fixes the bug, a test is added to ensure there is no regression.

@uwestoehr

If they need to compile their Julia module as an app, then you must provide this.

Are you joking? Julia developers are under no obligation whatsoever to cater to your specific needs.

You clearly have expectations that are completely disconnected from the realities of open source software development. Maybe these are expectations for commercial software, where you have contracts with a developer to provide specific services to you. Although, from my own experiences, commercial software often has just as many bugs, and development processes that are far less transparent.

P.S.: I definitely got sucked into “someone is wrong on the internet” here and in my previous post. Sorry this got a bit hot-headed. I appreciate @odow stepping in.

[odow]

Hi all, I don’t normally do this, but since most moderators are probably asleep I’m just stepping in to ask that we try to keep the discussion positive and refrain from some of the more personal comments.

It’s a Monday afternoon for me, but I assume for most others it’s late on a Sunday, which isn’t a great time for productive discussions. If you’re rushing for a reply, it might be best to come back tomorrow.

[Benny]

@Benny

if you want to invite properly focused discussions instead of a haphazard mixture or more likely avoidance, I suggest splitting your issues into separate topics

This topic went pretty much as I expected. There were plenty of good things to talk about, but the misguided association of several distinct issues devolved into disorganized, irritated discussion of only a fraction of them. That would happen even among cooler heads. Quotes and links freely traded among 2 other active topics are easily missed by readers, further compounding frustration. For example:

@uwestoehr

And I failed because it hardcodes Unix-special compiler settings.

This issue was attributed to the claim that cc is Linux-specific, specifically that it won’t use MinGW-w64 on Windows. Which is very strange to me because when I went to check my MinGW-w64 installation for the one time I dabbled in C, cc works, which I didn’t even know about. From Julia, run(`cc --version`) gave nearly the same output as run(`gcc --version`), so I tracked down the bin folder and found cc.exe and gcc.exe with identical file sizes. I have no clue what that means, but it was probably worth talking about in a dedicated topic before snowballing into lists of demands for the wrong audience. Again, I personally would like to learn!

[goerz]

@uwestoehr

I cannot find a publicly available checklist how releases are made, no QA policy, no policy how PRs are reviewed.

To bring this a little back on topic, here are some relevant links:

• Julia’s Release Process
• Making a Julia Release, JuliaCon 2023
• Julia CI (that’s the “QA”)
• CONTRIBUTING.md

The policies for reviewing PRs are just the normal workflows for GitHub-based development. I suppose those are more or less implicit. And yes, it is assumed that people contributing to Julia or the documentation of any package are familiar with GitHub. Maybe that’s a bit of a barrier, but I’m not really sure how I would improve on that.

Actual core contributors can probably give you more details about the development/release/QA process. Your desire to see these things explained is absolutely not unreasonable. But I would assure you that Julia has quite strenuous QA, even if not everything might be written down or might be immediately obvious to a newcomer.

[liuyxpp]

@uwestoehr

If Julia wants to be attractive for more users, it must please the users, and not tell them what they should do different. Julia must try to offer what users need. If they need to compile their Julia module as an app, then you must provide this.

I doubt this strong statement. Is Python or any other lang doing this? There must be a balance here. I don’t believe there is a single lang that satisfies everyone.

[gdalle]

Moderator here: when a point has already been made several times by yourself or other people, please refrain from piling on. The goal of this forum is also to entertain user feedback, even when we disagree.
@uwestoehr I think your comments would be much better received if they were phrased as suggestions, or even better, offers to help, instead of demands.

[xgdgsc]

The commercial product https://www.tongyuan.cc/release/syslab which offers GitHub - Suzhou-Tongyuan/syslab-deploy: Deploy your Syslab/Julia applications everywhere (Windows, Linux, macOS, Android, iOS) with Rust/Flutter. might be what you need.

[uwestoehr]

@gdalle

The goal of this forum is also to entertain user feedback, even when we disagree.

The point is what is the outcome of a discussion? What will/can/should be changed that e.g. the next release announcement is done after all new features are tested on the different OSes and if there are known bug reports, they are mentioned.

@goerz

o bring this a little back on topic, here are some relevant links:

Thanks. Before I started this thread, I had a look at CONTRIBUTION.md.
And my point is that in none of the documents you linked, there is a policy regarding the 3 major OSes, that e.g. PRs have to be reviewed with them in mind.

Maybe this thread helps that this aspect is added to the processes/protocols.

[goerz]

@uwestoehr

The point is what is the outcome of a discussion? What will/can/should be changed that e.g. the next release announcement is done after all new features are tested on the different OSes

Yeah, Discourse is probably not the most effective or most appropriate place for discussions like that. There are core developers that are active here, and they might draw their own conclusions from messages here. But generally, the development teams for particular projects set their own workflows. I think discussions about improvements in workflow would probably happen during the triage meetings, or in some of the channels on Slack. The list of supported platforms and their support status for Julia is quite explicit, and there is no question that the core developers are fully committed to that support. As for the “hard-coded” compilers in relation to --trim, I’m still not sure if that’s deliberate or not. Based on the other posts in the thread, that might just very well be the correct thing to do, even on Windows. Or, it might be that --trim is just so experimental that it doesn’t yet have tests covering non-Linux. A new release of Julia can absolutely contain new experimental features. You should not have any expectations about --trim in the upcoming Julia release. If something doesn’t work, you are free to file bug reports.

But the bottom line is that new feature are tested on the different OSes. Your experience is due to expectations that are fundamentally incompatible with how Julia is currently intended to work, trying to use a highly experimental feature, and, unfortunately, running into a bug in a third-party (albeit closely aligned) package.

@uwestoehr

in none of the documents you linked, there is a policy regarding the 3 major OSes, that e.g. PRs have to be reviewed with them in mind.

I should have followed my own links for Julia’s CI a bit further to get to julia-buildkite (the actual CI setup) and it’s documentation, which also contains supported_platforms.md. This is in addition to the more public table of supported platforms. PRs are most definitely evaluated with respect to these supported platforms. I don’t think that any fundamental changes in process are necessary. There is lots of work to be done in terms of new features (like static compilation), but those will happen on their own time, and should be quite well-supported within the existing processes.

[nhz2]

The current QA process for releases seems to me to be broken.

For example two weeks ago I reported a regression `IOBuffer` `skip` regression · Issue #57962 · JuliaLang/julia · GitHub

What is the process for getting this on the release milestone?

Another example is the new Sys.detectwsl function added in base: add `Sys.detectwsl()` by inkydragon · Pull Request #57069 · JuliaLang/julia · GitHub

The PR notes that this function is currently unable to be tested fully with the current CI system, yet the release notes and function documentation do not mark this as experimental.

Looking back at the 1.11 release, there are still opened issues about the broken --project=@script feature `--project=@script` does not activate script project if outside of the Project Directory · Issue #55069 · JuliaLang/julia · GitHub
Why didn’t this block the 1.11 release, and what is the process for quickly marking buggy new features as expirmental to avoid stalling a release?

[nsajko]

@nhz2

What is the process for getting this on the release milestone?

I think you have the necessary permissions on the JuliaLang Github yourself.

BTW, I don’t think I ever got instructions for how to use the triage powers, but basically I follow the Wikipedia rules: be bold, just do what you think is right, as long as you don’t wheel-war. By “wheel war” I mean “revert the action of someone else even when it’s clear the action was intentional and there is a difference of opinion”.

[Oscar_Smith]

What is the process for getting this on the release milestone?

Added.

[nhz2]

Think about this more: The current CI and PkgEval release process works amazingly well at preventing regressions of existing features, but it doesn’t check that all new features are well documented and tested.

Aside from the obvious of just having fewer complex new features in each release, here are three more specific things I think that can be improved.

Keeping the changelog up to date.

Currently, release announcements link to the tagged version of the changelog https://github.com/JuliaLang/julia/blob/v1.12.0-beta1/NEWS.md
This cannot be edited.
Instead, this could link to the master version of this file https://github.com/JuliaLang/julia/blob/master/HISTORY.md#julia-v112-release-notes which is hopefully easier to keep updated.

A new experimental section can be added to “HISTORY.md”. Currently, it is unclear which new features in the changelog can be relied on and which are more experimental, like the AnnotatedString or --trim features. Alternatively, experimental features could be removed from the changelog.

Making it easier for Windows users to contribute to docs.

CONTRIBUTING.md recommends running make docs before opening a PR. This might prevent people who are not existing Julia developers from contributing doc fixes, and I’m not sure if this even works on Windows. I think it makes sense for people who develop Julia programs for use on Windows to have a wiki or site like Julia on HPC clusters for Windows-specific tips and tricks.

Documenting more details of the GitHub workflow.

“CONTRIBUTING.md” already has information on this, but I think it would be good to explicitly document how, when, and by whom some of the special tags like “merge me”, “backport”, or milestones should be used.

[mbauman]

In addition to the links and references Michael provided above, I’d add a few things:

The release milestones on GitHub are generally used as a rough organizational tool; they aren’t the universe of what will or will not be in a release and they’re not promises, but they can be helpful… if for no other reason than to get an explicit decision on the release-blocking-ness of a particular issue.

There are — effectively — checklists for each and every single patch merged in — and that includes those targeted at releases. They’ve gotta pass a battery of CI machines for all supported (Tier 1) platforms, as well as a review process. Reviewers can easily add blocking tags (like needs tests, needs news, needs PkgEval, etc), and merging is prohibited until they’re resolved. The project is quite liberal with “triage” permissions — that’s what allows you to label issues like that — and it’s a great way to get involved and help ensure things don’t fall through the cracks. It can sometimes still be hard to get changes in. The vast majority of Julia packages similarly ensure testing and reviews.

In preparation for a release, a script gathers all the back-portable patches and ensures they can be merged cleanly. For example, here’s the ongoing work towards a second beta in the v1.12 line. Note that “rudimentary Windows support for juliac” is an ongoing task on master and is flagged to be back ported to v1.12. An outstanding (unmerged) patch flagged for backporting (like that one) will make it appear in that list and, at a minimum, will get the attention of the release manager and they can make a explicit decision if it should block the release or not.

You can also see the end result of running all the registered tests of all packages in the entire ecosystem (regardless of its version number or “production-readiness”) through PkgEval — and more importantly — the results of actually looking at it and identifying concrete issues. Kristoffer’s obviously not the only one who does this, but this is the best way I know to demonstrate a lot of such issues. He’ll also explicitly trigger PkgEval runs directly within the backporting pull request where you can easily follow along, too, e.g., Backports release 1.12 by KristofferC · Pull Request #57955 · JuliaLang/julia · GitHub. Now, yes, one limitation here is that PkgEval is Linux-only.

I think it’s worth pointing out that folks — some who haven’t even engaged here — have continued to improve things. There’s now a dedicated install page. There’s a pull request that looks to fix the PackageCompiler hang. Sometimes issues do fall through the cracks; bugs and regressions that don’t get accurately categorized (or bumped) can easily get lost in notification overflows. But they’re not lost forever! Friendly bumps can often work just as well — if not better — than demands and rants.

[mbauman]

@nhz2

For example two weeks ago I reported a regression IOBuffer skip regression · Issue #57962 · JuliaLang/julia · GitHub

What is the process for getting this on the release milestone?

Two weeks is a great turn time. Thanks for reporting it and even bigger thanks for the patch! The vast vast majority of patches that get into a release (and even many that get backported) never land on any milestone target. In fact, GitHub’s milestones simply don’t work for issues that have been addressed on master (but not the release branch) or PRs that have been merged to master but need a backport; it’s the label that’s important for that.

@nhz2

Another example is the new Sys.detectwsl function added in base: add Sys.detectwsl() by inkydragon · Pull Request #57069 · JuliaLang/julia · GitHub

The PR notes that this function is currently unable to be tested fully with the current CI system, yet the release notes and function documentation do not mark this as experimental.

WSL is a Tier 2 platform; we explicitly don’t promise that we test it or even that the tests may pass!

[uwestoehr]

The following is meant to be constructive, it is not to blame anybody, but to improve the release process for future releases:

For any software it is important that releases are

• either only made once there are no known regressions
• or if there are known regressions but a new release should/must nevertheless be made, the known regressions are listed in the release announcement together with a workaround and additional info (for example to stay for now with an older version or why a new release was nevertheless done)

Take the case of PackageCompiler. Since Julia 1.11 one cannot use more than one thread. This reduces the compilation speed a lot (for me by about 40%). The problem was already reported and verified in October. But since it was not mentioned in release notes, I run into that. Now I know, but it caused me hours to find out and thus of course also unnecessary frustration. Now I see that Julia 1.11.5 was released and despite this ongoing discussion where the regression was mentioned several times, it is not mentioned in the release notes and a new release was done at all.

Again, things are now as they are, but for Julia 1.11.6, 1.12.x it would be very good, if the release maintainers would adopt their checklist regarding known issues and regressions.

I know that some argument that PackageCompiler is not really part of Julia, but think about average users. They cannot know the internals of the Julia project structure. People like me chose Julia to use all features Julia advertises. And PackageCompiler is in many talks mentioned as one the key features for interoperability with other software.

[goerz]

@uwestoehr

For any software it is important that releases are […] made once there are no known regressions

I would agree with that in general. And I think most projects would and do. That is, they wouldn’t make releases with breaking tests. It seems like this issue wasn’t categorized as a regression, since it was a newly discovered bug and didn’t break any of the existing tests. Arguably it should have been. It’s probably a good idea to treat bug reports that say “this broke in release X but was working before” (a.k.a, a “regression”) with a bit more priority. But I’m also hesitant to demand that package developers, many of whom are academics writing software on the side, can never fall short of the highest possible standards of professional software engineering. That’s a recipe for open-source-burnout.

[mbauman]

The release process for patch (1.2.x) releases is slightly different than that for minor releases (1.y.0). The former are wayyy more conservative in what they include; everything must be straightforward bug-fixes. Patch releases are meant to be completely unobtrusive. They’re released much more frequently to help everyone avoid known and fixed bugs. Concretely, this means that the 1.11.6 patch release should not introduce any new regressions over 1.11.5; it’s no longer being compared against 1.10. Yes, it often fixes regressions over 1.10, but the time to find new regressions — and block releases — is in the much slower beta/rc cycle for the minor release. Once folks are using a particular minor release, it’s very much a good thing to get fixed bugs out as quickly as possible.

You keep talking about a checklist. There are multiple visible “checklists”: CI, PkgEval, backports PRs, issue labels, and milestones. The hardest part, though, is making sure the right line-items appear on them. But this is where open source can shine. There are multiple ways to make issues appear on one of those checklists, even without any special permissions:

• Add tests to Julia and the packages you use! Testing current working functionality is a great way to get involved in the process and broken tests are the absolute best way to ensure attention.
• Help review the 4k+ open issues on Julia itself, particularly the ones that are missing labels or haven’t been touched in a while. Make sure they have MWEs and still reproduce, and you can kindly ask for attention, too.
• Help bubble-up issues from the 10k registered packages to a JuliaLang/julia issue if it’s likely a problem in Julia itself. Everyone will have their own definition of what the most “core” or “fundamental” packages are; if a package you care about has a regression from 1.10 to 1.11, it’s probably better placed as a Julia issue. Open one! Link to the package issue and try to minimize the problem as best you can.
• Pop in on a release branch in preparation and ask if some issue in Julia or the ecosystem should be blocking.
• Test the ongoing v1.12 beta, report issues, and help ensure they’re categorized correctly! Now’s the best time

I’m sorry this was so frustrating for you. And I get it; burned time really stinks.

[Jake]

Having used Julia for a couple of years, and having downloaded, but not really used FreeCad, I think there is a difference in how QA should work, and that difference may be part of the frustration and talking at cross purposes expressed here. FreeCad has well defined functionality, much like the programs that the OP is trying to create. Julia is a general purpose programming language used mostly by scientists and engineers. I can see how the expectations around the development cycle can be different.

I do really like the suggestion of a wiki and the example given [above] (Introducing Julia - Wikibooks, open books for an open world) is nice and I didn’t know it existed. Having it or something similar more prominent in the Julia world would be an asset for Julia for new users and everyone.