Implementation Hiding Proposal

Internals & Design
1

Julia is now my favorite language by a wide margin, and aside from the minor quibble of pancakecaseidentifiers, i can hardly fault it.

There is one thing which does bother me however, and that is modules, import/export and visibility. I read a a thread here a while back while I was still a bit of a noob, which got a bit heated on the subject, but wanted to see if perhaps with time I might gain some perspective.

There were two general “camps” in the discussion, those who wanted
data hiding for software engineering reasons, and those opposed where the specter of Java privacy hell loomed large. I was somewhere in the middle.

The first thing which struck me coming to Julia was the verbosity of export lists, considering that you can just import X and completely override all of it. The second was that being able to do so was not at all POLS, and in my experience quite idiosyncratic to Julia. Because we have a REPL with tab completion the problem is exacerbated, when I write Module.<tab>, I want to see the public API not the implementation details. I’d like to be able to tell at a glance (i.e. a <tab>) that which I should be using and that which might disappear out from under me at the next release. Pouring over export lists is not “a glance”, but even if it were, Julia culture (e.g. Base) is such that some public API functions (convert, promote, etc) are not exported.

Julia is a “freedom” language though, I get it, we don’t want Java hell, however I think that a coarse grained module level overridable privacy mechanism would be a very good thing.

There are several possible routes here but I would like to propose a very simple one. Everything not exported is module private and does not show up on <tab>. Functions and objects not exported are assumed to be implementation details and are not to be trusted from an API consumption point of view, however any object is accessible via an explicit extract statement.

Something like:

# file: M.jl
module M
export foo
foo() = :foo  
bar() = :bar 
end

# file: importM.jl
import M
M.foo() # :foo
M.bar() # throws error

# file: extractM.jl
extract M
M.bar() # :bar
typeof(M) # ExtractedModule
extract M: bar
bar() # :bar

where: Module <: AbstractModule && ExtractedModule <: AbstractModule && Base isa ExtractedModule, and where using would remain unchanged, excepting in the form using M: bar where extract would have to be used instead.

I am cognizant that this proposal might be a really bad one for a number of technical reasons unbeknownst to me, and know that 0.7 is nearing, but am hoping that this issue might have some resolution, or even just one last good look/discussion before that occurs.

2 Likes
2

you can get the list of exported names with the method names(::Module), which I found out only recently.

It would be nice if the names(::Module) output could be displayed in column format as with the .tab

2 Likes
3

The problem with that is that the set of exported methods and the set of methods in the public API are not necessarily the same. It would be very cool to have some mechanism for telling tools (such as autocompletions in the REPL/Juno/Jupyter) which methods are public and which are not.
That said, I’m not a fan of actually enforcing “private” methods in any way.

2 Likes
4

The problem with that is that the set of exported methods and the set of methods in the public API are not necessarily the same.

Agreed, that is the problem I am trying to address, they should be the same.

Tools would simply reflect what either Module or ExtractedModule was exposing.

That said, I’m not a fan of actually enforcing “private” methods in any way.

Also agreed, hence extract.

5

To clarify: Code would have to be organized differently for sure. Public APIs intended for general consumption, but not exported would have to be put in separate modules. The exception to that rule, as alluded to in my original post could be Base where it could be an ExtratedModule, although it could also just be split into two modules, the auto-included one and the remaining (what is currently unexported) one. That would probably be the better route.

6

extract M: bar: How is this different from doing

import M: bar

? Could you elaborate please?

7

import M: bar would no longer work if bar were not exported. extract would have to be used in that case. this makes explicit the fact that an internal implementation is being used, and could hence break the code at a later date.

8

Got it.

9

Not sure if it is feasible nor efficient with some macro magic trick, but a possible idea could be “flagging” all function starting with underscore _ as private and not shown in REPL autocompletion.

2 Likes
10

Just to put it all together:

# file ExtractApocalypse.jl
module ExtractApocalypse
export ImPublicToo, callme
const IMPLEMENTATION_DETAIL = :no_worries_mate
callme() = IMPLEMENTATION_DETAIL
 
# we put public APIs we do not want to export from main module in
# other modules, possibly as in this case submodules
module ImPublicToo
export callmetoo
callmetoo() = :your_good
end
end

# using example
using ExtractApocalypse
callme() # :no_worries_mate
ImPublicToo.callmetoo() # :your_good
IMPLEMENTATION_DETAIL # error
ExtractApocalypse.IMPLEMENTATION_DETAIL # error
using ExtractApocalypse: IMPLEMENTATION_DETAIL # error

# import example
import ExtractApocalypse
ExtractApocalypse.IMPLEMENTATION_DETAIL # error
import ExtractApocalypse: IMPLEMENTATION_DETAIL # error

# extract example
extract ExtractApocalypse
ExtractApocalypse.IMPLEMENTATION_DETAIL # :no_worries_mate
extract ExtractApocalypse: IMPLEMENTATION_DETAIL
IMPLEMENTATION_DETAIL #  :no_worries_mate
1 Like
11

I definitely appreciate the problem you are describing here, but I can’t help but think that making a change to the language (apart from possibly some convenient function in Base) would be like lighting a cigar with a cruise missile (i.e. overkill).

This seems like mainly a documentation issue. Perhaps a standard way of flagging public functions that are not exported in the doc-strings? Another, somewhat more drastic possibility would be to provide a macro that allows one to specify that a non-exported function should appear in the output of names.

I know that Documenter.jl already has something like this, at least for the purposes of documentation.

1 Like
12

The names method already has a Boolean option called all for that

help?> names
search: names fieldnames dirname tempname fullname basename TypeName fieldname

  names(x::Module, all::Bool=false, imported::Bool=false)

  Get an array of the names exported by a Module, excluding deprecated names.
  If all is true, then the list also includes non-exported names defined in
  the module, deprecated names, and compiler-generated names. If imported is
  true, then names explicitly imported from other modules are also included.

  As a special case, all names defined in Main are considered "exported",
  since it is not idiomatic to explicitly export names from Main.
13

That’s something different. I think what’s bothering @polypus74 is that there is a little bit of an unspoken assumption in Julia that all and only exported functions are “public”. However, for some packages it doesn’t really make sense to export, and for those packages, it can be a little bit more difficult to reckon which of their functions one should consider “public”. names with all=true just shows you everything.

Like I said, I think this is mainly a documentation issue, but it’s a legitimate one that I’ve noticed before.

1 Like
14

This is definitely an issue I have pondered before as well. I think what people are getting at is that there should really be 3 states instead of import/export being a binary choice.

  • export keyword declaration makes a method exported as usual
  • no keyword declaration makes it a function in the usual sense that is not exported
  • hide keyword could be a new declaration that explicitly marks a method as hidden so that it does not show up externally from the module

This might be what is needed, since some names in a module are completely internal and don’t need to be exposed to the outside world at all. However, it should be an explicit declaration, just like an export declaration, so maybe something like a hide or hidden declaration for names is needed ? e.g.

module
    export external_method
    hide internal_method
end

this wouldn’t break anything, it would simply be a new feature.

3 Likes
15

Another simple idea is that by default all functions which are either exported or have doc strings are considered “public” (mainly I think all this would mean is that they show up in names by default). Then, if you want to document private functions you can do something like

"""#private
This is some documentation.
"""
func() = dostuff()

I don’t know if it’s really such a good idea to have names looking at documentation strings but again I think it’s unfortunate when documentation issues leak into the design of the language (or libraries) itself.

1 Like
16

@chakravala: yes hide or just the more traditional private could be used, only you need an escape hatch. maybe:

module M
private foo
foo() = ...
end

foo = extract(M, :foo)

The main problem with that however is that within a single package you would really like to have a less cumbersome way of using private variables of submodules in the same package. extract M: foo is more in keeping with using and import, and within a single package could be thought of as completely equivalent/benign as it is not referencing a dependency.

17

A possible smooth transition path for my proposal would be to drop extract (which is a bit unwieldy/uncommon a name anyway) keep import as is (i.e. it is equivalent to extract), and add a require Mod: var keyword (borrowing from Ruby), which would play the role of import in my proposal. People could then just transition over in time replacing their imports with requires.

EDIT: using however would change and in the case of accesses like Mod.unexported, would require a previous import, which would mean that import would have to somehow “augment” any previously using’d or require’d module, which could present an implementation problem?

18

I’m in favour of this idea, this is basically what’s done in Python:

_single_leading_underscore: weak “internal use” indicator. E.g. from M import * does not import objects whose name starts with an underscore.

and many Perl coding standards follow it too. It’s a clear signal about whether the name is internal or public, and it sticks with the name itself which makes it obvious at the point of usage that an internal entity is being used (rather than at a private declaration in the module and a different importing keyword at the top of the file).

Also, at least for me, a simple _ added at the beginning makes it clear that it’s just a weak indicator of privacy, while having a private or similar keyword seems to make stronger promises that we don’t want to enforce in Julia.

3 Likes
19

FYI that’s a first: "lighting a cigar with a cruise missile" - Google Search :wink:

3 Likes
20

Funny that you bring this up today, I’ve been sweating through metaprogramming to handle my own problems with import/using/export.

My solution (as long as I can get the macros to work out) is to have an API package (need to bikeshed on the name), that helps you set up a list of functions that are part of the public API that might be extended, a list functions that you don’t want extended as well as types, modules and other things that cannot be extended,
a list of functions that may need to be extended, but are not part of the public API (they are really part of a development API), another list for development functions not to be extended, and finally a list of functions that need to be extended from Base (for example, start, done, next if your API includes iterators).

Here is how I think it would be used in practice:

using API
@api extend StrAPI, CharSetEncoding, Chars
...
@api public_methods string_distance
@api public_defines   Str, is_multi

With this, it’s easy to find out the public API, but you normally would not export anything, the macro would use either import or using to pull in only the names that you need, either for simply using the package, or for building things “on top” of it (extending the functions).
This is saving me from export / ambiguity hell as I am trying to refactor my Strs.jl package.

1 Like