Exactly - it could be something for v2, or 3 or never. But it needs to be brought up and discussed in order to get there.
So what about just pushing for @public like @dilumaltthge proposed above? So in the case it is not exported but public you just annotate it with @public. The set of public API functions are all functions that are either exported or marked with @public.
But I mention could be done using comments instead of docstrings. I yet prefer @dilumaluthge solution, but what you pointed is not a strict blocker.
Your initial post was about a field that was neither documented nor exported. By your last post, I do get you now agree that there was no proxy that indicated it should be accessed, right?
3 Likes
Documentation can be distinguished between general (public) and developer docs (private). This is the current status as far as I understand.
And just to be clear, I think the @public proposal of @dilumaluthge is great in the fact that it doesn’t break current usage of private parts of packages (which in my eyes is one of the great advantages of Julia), but allows better tooling and workflow. 
3 Likes
I’m not a Python developer, but it looks like these are not conventions, but actually enforced access modifiers?
Not really. The double underscored “private” method is a naming convention that the interpreter also happens to do some name mangling. But you can still access the the field easily, no need for reflection (as in Java) or dark magic (as in Rust). And TBF, it’s not that common, ppl use the simpler single underscore convention way more often. Python’s philosophy here is “we’re all adults”.
The problem with this convention for Julia is that it makes members default to public. This is not the status quo and would break current semantics.
I’m also in favor of a non-enforced @public annotation, and let linters/REPL warn you of unexpected private usage. It would require way less changes in existing code to conform.
1 Like
Even with current practice, just having a docstring does not imply that something is part of a package’s interface. Julia programmers are free to document everything with docstrings, and even include it in the package manual under a section titled “Internals” or similar.
Personally, I find export to be a perfectly adequate marker for advertising the API. There are only two issues:
-
tooling (eg completion), which could be improved (eg user needs to press TAB twice to complete to an non-exported name)
-
name clashes with
using SomePackage, but those can be resolved with the usual way
That said, can you please clarify the what made you think that Base.modules_warned_for was public in the first place? It was neither exported nor documented.
It is not clear to me how any of the proposals in this discussion would have made it more clear.
5 Likes
No wonder why JavaScript community is moving to TypeScript: https://www.typescriptlang.org/docs/handbook/classes.html#public-private-and-protected-modifiers
much better on code readability. Still doesn’t replace private public protected but much cleaner than anything not documented in manual is private/internal and subjected to change
Do you think it is fair to compare TypeScript with Julia? I’m personally more interested in Julia making most of LLVM performance instead of supporting large scale software development, but that is only a data point.
Or to be rejected.
The disussion so far seems to show that chances are bad that we get there were you want us to be.
I agree that we are missing a mechanism to determine API programatically, as @pfitzseb pointed out. But this is not the same as OP wishes. It is a complete different issue. I don’t think those two issues should be mixed together. My opinion is Julia doesn’t need something like @public/@private for users or package developers.
I didn’t compare I replied to non-jedi’s comment about popular language JS/Python not having access modifiers.
I like how JavaScript community neatly documents APIs using comments/doc strings
/**
* Array of CLF month names.
* @private
*/
var CLF_MONTH = [
'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun',
'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'
]
/**
* Default log buffer duration.
* @private
*/
var DEFAULT_BUFFER_DURATION = 1000
/**
* Create a logger middleware.
*
* @public
* @param {String|Function} format
* @param {Object} [options]
* @return {Function} middleware
*/
function morgan (format, options) {
var fmt = format
var opts = options || {}
1 Like
Just for your information, this change breaks the MVC App generator from Genie so maybe it wasn’t such a good idea to silently introduce a breaking change.
To quote Linus Torvalds “If it’s a bug and a lot of people are using it, then it’s a feature, not a bug” and “don’t mess with user-space” or something along these lines.
1 Like
Is there an issue in Genie.jl about this yet?
1 Like
Yes, I’m aware. That’s why this thread was created after all.
I really don’t know what to tell you, other than pointing to the discussion that has already happened in the past 50+ replies. Julia is not the linux kernel. Even the linux kernel breaks user space. Julia has a different policy than the linux kernel about API. Yes I do wish it was a different policy (still not the one the linux kernel uses), but that’s not for me to decide now, is it? Please don’t shoot the messenger.
Not that I know of - the creator of the package opened this thread originally, so presumably they’re already aware of this, even without an issue. Since this is only used in a single place by Genie, seems like it could be solved by using a different mechanism (or removing the offending line entirely) and/or hiding the behavior behind a VERSION check.
1 Like
It didn’t make me personally think it was public. The feature was contributed by a dev as an OSS PR. If the tooling does not enforce the rules, the tests pass. So some very busy human needs to actually go and check that every function that is used in every PR is exported. Is this scalable/sane (rhetorical question)?
3 Likes
Since this is only used in a single place by Genie, seems like it could be solved by using a different mechanism (or removing the offending line entirely) and/or hiding the behavior behind a
VERSIONcheck.
Yes, there is a workaround, but working around a breaking change is not the issue. It’s just the fact that the change was introduced without any announcement, which makes it hard for package maintainers to keep their packages up to date. There is no need to get defensive about this, a small hint in the upcoming 1.8 changes would have been enough. And of course something to keep the private API really private in the long run would help.
Concrete answer: yes.
As soon as you test for the new version of Julia, tests should break and it will be caught. It is good practice to run CI on nightly anyway, then you will be notified very early.
Tests for registered packages are routinely run before each release. See
2 Likes
I really don’t know how to word this better, but to me that sounds like that PR should not have been merged as it was, if “Don’t rely on undocumented, non-API Base internals” is a critical policy for your package. It is not the responsibility of Base to ensure your package is using the API correctly or doesn’t access internals it shouldn’t access.
Again, it’s absolutely fine if you decide that the function is useful to you and you want it. You do assume responsibility for your uses of it though, since there is never and has never been a stability guarantee for internal functions.
Likewise, Base cannot go around and vet every package in existence to check whether they conform to the documented API or not.
I want to make this absolutely, perfectly, 100% clear: This was not a breaking change. Just because some internal function doesn’t exist anymore or changes behavior, does not automatically force the change to be a breaking one. Breaking changes only ever relate to changes in documented API.
As I said above, it is NOT feasible for Base to go around and check every single package in existence on every single PR to Base for any potential breakage. Not only would that be a huge timesink in terms of “time until a PR to Base gets merged”, it would also constitute dealing with already broken packages and having to review them on any change to Base. This also doesn’t work for PRs to Base that require subsequent follow up PRs or require some other work from some other PR that hasn’t been merged yet, but is otherwise complete to base new PRs off of.
This is not an option and is why API as a concept is distinct from “the functions I can reach syntactically”. Since the function in question is not part of the documented API, the change was not breaking, because it did not break any existing uses of the documented API.
On top of this, every release for julia has a list of included commits, automatically generated and linked in every announcement post for that version (here’s the announcement and here’s the list of commits for 1.8-rc1). It is not feasible to write a blurb in NEWS.md or some other document for every single removed/changed/added internal function - that list would very quickly become much too large to be practical to scan or read through, when a single CI run of a package in the new version can already detect that change perfectly fine and report it to the people most interested in that specific change: the maintainers of the package.
Also, as @Tamas_Papp mentions, PkgEval is already run for every package in existence before a release and if your package is found to break due to a change in API, you will get notified/the change will be reverted (depending on severity). In this case, the package is relying on an internal, undocumented function that is not part of the julia Base API, so the break is on the package maintainers to fix for relying on the function in the first place.
12 Likes
Reading through this thread, it sounds like 3+ separate things are being mixed up. The meaning of private/public w.r.t. APIs is not the same private/public w.r.t. access modifiers, and the public API of a software company, application, or service is not the same concept as the public API of the Julia language or a package.
To address the original question without all this distracting fluff, it just sounds like there’s a need for a method to be stable, but where is the line and who draws it? It’d be ridiculous if every method was documented to be stable. Development would freeze, and the documentation would be bloated by helper methods (_getindex, _broadcast_getindex, _broadcast_getindex_evalf) that users couldn’t care less about.
So it really comes down to who decides what methods are stable, and the developers who maintain the code and wrote the user documentation probably have more of a right than a user. If a developer decides a method will be stable, they’ll document it. It’s only fair that if a user wants to make the same decision, they should also contribute a small bit to the standard so everyone else is also on board. This would be far less drastic than introducing access modifiers, which by the way aren’t for documenting stability. (Some) object-oriented languages use them (in different ways) for encapsulation of components of classes, which do not exist in Julia.
2 Likes
Unfortunately you’re incorrect.
Genie does run tests on nightly but either Julia 1.8 was not nightly back then, or the feature was not yet removed.