Looks like Base.modules_warned_for no longer exists in 1.8? I can’t find it mentioned in the news. What’s the status please and what’s the recommended way for replacing it?
1 Like
That was an unexported, undocumented internal variable - they’re not guaranteed to be around in any version. It got removed here:
5 Likes
Thank you for the info.
1/ it’s not the first time I see the “undocumented internal variable” as an excuse for breaking changes in minor releases. IMO this is bad API design and reminds me of the Windows API disastrous approach decades ago. Anybody knows if access modifiers (at least public and private) are anywhere on the map?
2/ Is this going to be documented in the NEWS as a breaking change in v1.8?
2 Likes
This never was API. This was an internal variable used as internal storage. Just because you can access it doesn’t make it API. Nothing stops you from figuring out some internal function pointer in the Linux kernel and just calling that in your code. That doesn’t make it safe or reliable for you to do.
The general rule of thumb for julia is “If it’s in the manual, it’s API”. Anything beyond that is internal & subject to change.
No. Only changes to exported/documented functionality or documented new features (i.e., API) usually “make” NEWS.md. As this was never exported/documented, it’s neither a breaking change nor one that requires a deprecation.
21 Likes
Sorry, but that’s clearly not a good practice.
1/ Empirically it’s been demonstrated by 6-7 decades of computer science with the wide introduction of access modifiers.
2/ the fact that something is not exported does not make it private in Julia (not by convention, nor by the compiler).
3/ the fact that is not documented also doesn’t make it private - it strictly makes it “undocumented” which means it just lacks documentation at that time.
2 Likes
Don’t get me wrong please - I see your point as the current state of things. But it’s far from ideal and I would like to understand what is the wider opinion and the general thinking for the future.
2 Likes
My understanding was always the opposite. My experience with Julia is that if something is not documented means that it is private. If it becomes documented in the future then it will become public.
14 Likes
Look, I get that this caused you some breakage. However, for Base and stdlibs, the policy has been “if it’s in the manual it’s API, if not it’s not” for the longest time now. As such, for “Julia” the language this is the convention.
The number of packages that have non-exported, internal functions IMO (and I’m willing to say the numbers would back this up) vastly outnumber those that say “access everything you can, it’s all API”. There are a number of packages that willingly don’t export (part of) their API, but AFAIK they’re also explicitly saying “use it like this” or avoiding a namespace conflict and not relying on the user to just guess what is API.
The trouble with treating any observable change as API breakage is that you either can’t change anything anymore or you willingly break everything all the time. This isn’t any different in languages with access modifiers, as that policy would also cover behavioral changes, i.e. where a method takes more/less time, chooses a different internal struct layout that is smaller in size while achieving the same goal, etc. At some point, you have to draw a line where you say “no, this isn’t breaking”, and for julia Base that is (and has been, historically) here.
In regards to access modifiers - no, there are no planned changes or new features in that regard in the works as far as I’m aware. It’s generally seen as a benefit to be able to access internal fields of structs if so required (a library author can’t know how their code will be used after all), but this should ALWAYS come with the understanding that it’s not a reliable way to do whatever you want to do.
11 Likes
Thanks for the thoughtful response. Let me explain why I think that this approach is indefensible: private APIs need to be documented too, as a way to improve software quality and for the sanity of developers, maintainers and contributors.
The lack of documentation is a negative and can not be converted into a virtue that signals private access. The fact that Julia makes non-documentation a design pattern for its internals is very concerning from a code quality perspective. The fact that we want to encourage this to be adopted by the whole package ecosystem is a recipe for disaster.
5 Likes
I agree it is desirable. However, note that there is a difference between developers’ documentation and users’ documentation. At least my understanding is that what is included in users’ documentation is to be considered to be public.
In particular (me experience as DataFrames.jl maintainer):
- we do have documentation for some private functions; just this documentation is not included in the manual for users (it is explicitly excluded in https://github.com/JuliaData/DataFrames.jl/blob/main/docs/src/lib/internals.md and this page is skipped when documentation is generated)
- you can always document private functionality and explicitly tell your users that this is internal; what is important is the rule that:
- if something is not part of users’ documentation it is private
- some private parts can be included in users’ documentation but they must be explicitly documented as such
Having said that I must agree that the rules we have now do not encourage documentation of the internal functions enough (i.e. although it is possible to do it without a problem we often do not do it and ideally we should).
12 Likes
FWIW, “lack of documentation” doesn’t mean “doesn’t have a docstring”. It means that everything not in a sufficiently API-like looking section in the docs is to be considered private.
That’s horrible and I hate it, but it is the status quo. I’d love something more straightforward (a proper annotation would also allow us to hide non-public functions in the various auto-complete features everywhere).
17 Likes
Please don’t get me wrong, I do agree that I think internals should be documented! I also think that existence of documentation should not equal API and made effectively the same complaint with possible solutions in a comment on a PR. Granted, the place for this discussion was not right, but if you want to continue the train of thought I had there, I welcome it! I was only responding & describing the status quo after all.
Though I must preface this by saying that I wouldn’t want this to end up with compiler-enforced access barriers. Documentation wise, yes absolutely, but please don’t prevent me from accessing internals if I choose to take the risk of it changing in the next version - that’s a maintenance burden a developer willingly takes on by accessing internals.
8 Likes
1/ I totally agree, access modifiers should be discussed more broadly at community level, in a public forum - we need at least some sort of solution to the “non-documented means private” problem.
2/ because de facto everything is public, the maintainers should be a lot more cautious when removing features. Looking at the issue where this decision was taken, it looks like this could’ve been left until v2 (but this is my opinion). This is the problem with blurry boundaries - if you can’t express it in an algorithm that the compiler can understand (or there isn’t even any consensus between humans at all!) then it’s too subjective to apply consistently.
3/ there are many languages using access modifiers that provide a lot flexibility (Ruby comes to mind, but also a strict language like C# provides many such mechanisms).
Edit 1:
@Sukera btw, you make great points in that PR - and also happy to see that @kristoffer.carlsson noticed the same problem.
2 Likes
Since I don’t see this earlier in the thread, I will add that my understanding (for Base in particular, but in practice also for the ecosystem) is that:
- The public API is the API (private methods are internal, and not considered part of the API)
- The (public) API is defined by whether or not a name is
exported (unless explicitly documented otherwise, e.g. a name that is intentionally not exported to avoid name conflicts, but is documented to be part of the public API) - Semver only applies to the public API
Documentation is technically orthogonal to whether something is part of the API or not, but in practice for Base Julia, internal/unexported names are much less likely to be documented. This contributes to a common understanding that undocumented names are private, while documented ones are public, when in reality, the documentation existence is more of a byproduct of the rule/goal that all public names must be documented.
6 Likes
The problem is, that’s not the case. In the link shared above https://github.com/JuliaLang/julia/pull/42863, @kristoffer.carlsson mentions a few public APIs that are not exported. I could also add the famous Base.@kwdef and I’m sure there’s many more.
As the saying goes
In Theory There Is No Difference Between Theory and Practice , But In Practice There Is.
1 Like
For what it’s worth, the JuMP style guide recommends underscore prefixes for internal functions. Given that Julia lacks public/private access modifiers (not suggesting that it needs them), it’s useful for callers to have an obvious visual signal as to whether the symbol is internal or external. I agree that the presence of nice documentation per se isn’t a good discriminator between public and private APIs, because private APIs should be documented as well.
8 Likes
Turns out I was wrong, at least with respect to Base. FAQ: Public API states that the public API is defined by the criteria of “documented in the manual, and not marked as unstable”. The stated SemVer process in Julia:
Minor releases are also where significant refactorings of internals go, since we should only be refactoring to the extent that is necessary for fixing bugs in patch releases. This means that if you’re relying on some internal Julia stuff that’s not public, your code might break in a minor release. This is allowed according to SemVer since the change isn’t to a public API—so technically it can break at any time; but we will avoid this in patch releases, so minor releases will be where you have to watch out if you rely on internals somehow. [Emphasis added]
So removing Base.modules_warned_for is not considered a breaking change since it wasn’t documented in the Julia manual.
Both public APIs mentioned in the linked PR (Meta.parse and Profile.print) follow the public API rule (documentation in the manual), and are intentionally not exported to avoid name conflicts with other functions, and I would argue that Base.@kwdef is a good candidate for being added to the public API due to its widespread use.
I don’t have significant experience in languages with public/private interfaces, so I don’t understand what value that would offer over the existing export mechanism in Julia. What would you expect from a public/private mechanism that exporting isn’t able to achieve?
2 Likes
At least I can contribute an example
ForwardDiff.gradient(f, x)
ReverseDiff.gradient(f, x)
Zygote.gradient(f, x)
But this is not a matter of export, to support this you’ll probably need something more akin to namespaces…
What does this illustrate? Julia already has namespaces, in form of modules:
- Modules are separate namespaces, each introducing a new global scope.
The need to qualify gradient with their respective module is precisely because these implementations share a name, but not behavior (and they don’t share method tables, so they’re not the same function!). A public/private distinction wouldn’t add anything here, as all three are intended as public API.
2 Likes
It illustrates that differentiating API vs. private functions by export alone isn’t enough IMHO, and the differentiation by ‘is documented’ is slightly artificial…
2 Likes