25 Likes
FWIW, compiler enforced access modified are a Bad Idea. That said, I would like to see some more formal way package designers could indicate users that something is safe to use, or if it’s internal.
I think module level opt-in access “hints” would be a better approach. The module could add a “default visibility public” or “default visibility private” attribute to indicate the semantics of non-exported members, overridable as needed.
Usage of a private member would trigger warnings on first use, that could be silenced, similar to Python warnings.
I like the opt-in approach as it does not force package developers to make “hard” decisions ahead of time and won’t break anything in terms of current compatibility.
3 Likes
Can you please share resources that explain why? It’s a surprising take considering that all top used programming languages do it?
I’m fairly certain neither python nor javascript have compiler-enforced private methods/attributes, and depending on which list you look at, those are the two most popular languages right now. To be clear, I don’t see this as evidence in either direction for compile-enforced privacy as good or bad, but it’s inaccurate to call the lack novel.
For what it’s worth, private fields have been discussed several times in julia repo issues/juleps. This github issue is probably a good starting point with links to the other relevant issues/juleps.
7 Likes
I would argue that they’re (a bad idea) in the specific case of Julia. There a a some packages that access private parts of another packages/Base/Pkg to provide some functionality that the original package does not, and otherwise there is no public interface to make such functionality available, an example of this that comes to mind is TestEnv.jl.
The fact that you can always access “private” parts of modules allow you to experiment and try to do new things more easily in a lot of cases, or even for it to be possible.
8 Likes
I will abstain from commenting on the design of languages like JavaScript and Python.
The exact implementation is less important now. But we certainly need a solution to unambiguously designate private APIs in a way that can be programatically checked (if not enforced). I think most people are in agreement with this?
7 Likes
I am not in agreement — I think you are approaching this problem the wrong way.
A well-designed package should document what its API is, and then conform to SemVer. A convention that some Julia packages follow is eg exporting and/or documenting their API, but it is basically up to the programmer as long as things are clear.
Everything else is private/internal by default and should not be relied on outside the package, unless you are willing to deal with the consequences. Yes, people use Julia internals occasionally as a workaround, but then make sure to catch changes with unit tests and then it is their responsibility to fix things. Julia maintainers usually run PkgEval before releases and you get a courtesy warning about these things.
A “private API” is an oxymoron: it is just internal code organization.
13 Likes
I don’t understand what a “private API” is meant to be. But I do agree with the proposal of facilitating a programatical way of checking what is API and what is not. An easy way of doing it would be with some kind of key tag in the docstring. Then a function could be made that looks into a given module, and tells what objects have a docstring with that key (assuming that undocumented objects are never API). Exported objects might be optionally added to the list, regarldess on how they are documented.
1 Like
The issue with this approach is that it’s really hard to determine a package’s API programatically, which makes for bad tooling, which makes for inefficient workflows. If you write YourPackage.foo<tab> in the REPL or an editor with auto-completions, it would certainly be nice to only get the public foobar and not the private foobaz (or at least be notified that foobaz is private). Having to check Documenter docs for each function really doesnt’t make for a fun workflow.
16 Likes
A private API is not an oxymoron:
1 Like
In the article you link, a “private” API is private in the business sense, but it isn’t from the perspective of programmers who write code calling it (eg within the company). In other words, if one is using an API, it is no longer “private”.
(Also, I am not sure what the point is in sidetracking this discussion with an article of APIs as a service.)
I usually just retrieve docstrings with ? in the REPL. That said, it would be great if more packages had a module docstring that just outlines the API. A great thing about Julia is that multiple dispatch allows keeping APIs relatively small in terms of vocabulary.
That said, I understand the problem with completions — it cannot be determined from the context whether the user wants completion for exported keys or all keys. Types have propertynames, but there is no similar functionality for modules.
2 Likes
Well, getting completions for public bindings seems like a sane default – there’s just no way to determine which those are. Exported bindings are of course also public, but we’re missing some kind of @public decorator for unexported bindings (think JSON.parse etc).
3 Likes
I don’t want to sidetrack the discussion, but you keep insisting with ad-hoc definitions of “API” to make the point that the idea of private APIs does not exist. That is not so, it’s a common concept in CS - ex Additional class libraries and APIs - .NET Framework | Microsoft Docs
The linked articles clearly define an API: “Application program interfaces (APIs) are code that enables the communication between two software programs. APIs determine how a developer requests services from an operating system or application”. In our case Julia is the application and these APIs can be public, private or otherwise.
Overall I’m surprised that highly intelligent people are considering unique hacks (using documentation) for solutions to problems that have been analysed and solved in Computer Science for decades. This is a design problem that should be approached by researching and evaluating best solutions and practices. Sure, by all means, let’s come up with a new/better solution, but it should be backed by data, not “I think” and “I don’t want/like”.
I would like to understand how such language decisions can be discussed in a wider forum and with a more methodical approach.
5 Likes
One way to have a open and methodical discussion could maybe be to avoid this kind of formulations, and also to provide the alluded analysis and solution when making such claims.
For me it has never really been a problem with how Julia handles it, I have used non-API methods but then been aware that it might break. And I quite like to be allowed to do that to be able to explore, so I think I’m also against compiler enforced API boundaries.
But having something that can make the exploration and documentation of APIs easier sounds really great. Though, I’m not sure that is more important than anything else the core contributors are working on, so that is also an issue. But starting to discuss what it should entail and how it can be built is always a start, and then someone with the time and interest can maybe take it on.
2 Likes
They may of course exist, but are come with no guarantees for the user, especially not when it comes to SemVer.
So, for example, a Julia package may have an internal/private API that is well-documented, mature and robust, but as far as SemVer is concerned it can completely change or even disappear from one release to the next.
I am still not sure what the “problem” is here, let alone what the solution would be. My understanding is that you used an internal variable which disappeared from one release to the next; I don’t see any violation of SemVer.
I am assuming that you have an actual problem to solve and your previous solution relied on this variable. I think it would be more productive to discuss that problem instead. YMMV.
6 Likes
Python has the nice and simple convention that anything starting with an underscore is considered internal.
This works well with tab completion: x. completes to x.foo but not x._foo and x._ completes to x._foo. So by default only “API” (non-internals) are completed but you can get at internals by explicitly having the leading underscore.
3 Likes
Related discussions and packages:
9 Likes
I agree with the sentiment of the community in general.
- If it is exported it is public.
- If it is documented it is public, unless the documentation itself indicates it is internal (either by putting it in a developers’ section or just saying it is internal).
- Otherwise, it is private.
I sincerely roll my eyes to the idea that Software Engineering suggesting an organization pattern is the same as “solved in Computer Science for decades”. And while I like the spirit of the “backed by data” approach, what defines the path Julia will take are the Core developers (taking or not into account what the community has to say), or anybody crazy enough to fork Julia an maintain a version with their own changes. Nothing else. If the core developers and the community do not like the idea, and especially if you are not gonna implement a PR yourself for the idea to be tested, do not matter how rational the idea sounds to you, it will not take off.
My personal arguments in favor of the current model and against a change that include @public/@private modifiers:
- I do not want that Julia to have anything that enforces the private API to remain private. I want to be able to experiment with the private API of external packages (or Base) during development, and I do want the freedom to publish a version of my package that relies in other packages internals, and it will be entirely safe because I used Project.toml to only allow using the specific version of these external dependencies I tested. Such is common and necessary for me as a PhD student. I may need to use some package internals to do something, and while a PR of mine to add/expose such functionality is being evaluated, I need to be able to submit a paper following a deadline where I can point to a reproducible version of my experiments. Happened more than one time. At that time, I would have needed to abandon Julia if I could not do things this way. And while working in an Amazon Internship as Applied Scientist I had to do the same be able to provide a proof-of-concept by a deadline. After I was able to update my code to use the patched version of the package as my PR was accepted.
- Now let us consider that the question is not enforcing private/public semantics anymore, but letting a human differentiate between the two. The current system already does that. Every function is considered private by default (what is the ideal), because it is not exported nor documented by default. The author has to do an additional conscious effort to export and/or document it. If the function/variable/constant is not documented you should not even know/assume how it works or how it should be used. Any misuse of such piece of code is, for me at least, entirely due to the complete recklessness of the user.
- Now let us consider a final question, the only one I agree that probably needs improvement, how to have code differentiate between public and private API so productivity tools act accordingly? (i.e., do not suggest a private method in the autocomplete, for example.) For now, the only thing that can be used is checking if the symbol is both exported and has documentation. I am assuming that if an package developer documents and exports an internal object then the blame is on them. However, it is not an ideal solution, because sometimes a public function is just documented but not exported (for example, because it clashes with some function name from Base). And as private functions should be documented too, only having documentation is not enough of a criteria. There is a multitude of solutions that I will list, but all have the same problem, they would need to be adopted by the community, otherwise they have no effect. Any solution creating a default visibility will either need the users to annotate what is private and suffer from the adoption problem, or it will be breaking (because it would change the default visibility to not be public), besides the clash with my first point. Some proposals are:
- Have every internal object start with an underscore.
- Have a macro to annotate the visibility of the objects.
- Use the current system, but for documented internal objects use comments instead of docstrings.
- Use the current system, but for documented internal objects require the docstring to indicate (in a machine-friendly way) that the object is supposed to be private.
11 Likes
I’m not a Python developer, but it looks like these are not conventions, but actually enforced access modifiers? https://www.studytonight.com/python/access-modifier-python
Quote from linked article:
If we want to access the private member variable, we will get an error.
We’re going around in circles, this has been commented at the beginning of the thread: documentation is not a good proxy for access (private APIs need to be documented too). Also, exporting is not an accurate proxy for access (there are public APIs which are not exported - again, check comments above).
1 Like