Namespacing of methods

Internals & Design
1

After a couple of months of using Julia, I’m still a little confused about how methods interact with packages/namespaces. That is, situations where a package adds additional methods to a function originally defined in another package.

If I have a package A that defines (and exports) func and another package B that defines additional methods for func, it seems like doing using A and using B in the REPL, I get the combined methods from A and B. This does not seem to depend on whether B also exports func. Is that officially how things are supposed to work: If I’m using A, do all methods for func defined in other packages that I might use directly or indirectly get merged in automatically? With “indirect” I mean if I’m using C and the package C has a uses B, do I get the methods that B defines for A.func automatically?

Is there any way to use (parts of) B but not use the methods that B defines for A.func? One might expect that if I do using A and using B: func2 (where func2 is some unrelated function), I wouldn’t get B’s method for A.func, but in fact I do.

Second, while an explicit export of func in B doesn’t seem to affect whether B’s method for A.func gets used or not, it does seem to have some effect when I’m not directly using the package A that originally defines func. If my package B looks like this:

module B
import A
A.func(i::Int) = i
end

and I’m only using B in the REPL but not using A, then neither func not B.func is available. However, if my package B reads

module B
import A: func
func(i::Int) = i
end

then using B (without using A) in the REPL does give me access to B.func, and B.func includes both the methods from A and from B.

If I further export func, that is,

module B
import A: func
export func
func(i::Int) = i
end

then using B in the REPL (again without using A), gives me access to both func and B.func, again using the methods from both A and B.

Also, in that case, if I’m both using A and using B explicitly in the REPL, I can address func as func, A.func, or B.func, with all three being completely equivalent.

Lastly, if I define the package B as

module B
import A
export func
A.func(i::Int) = i
end

I don’t get any complaints from Julia, but the REPL autocompletion thinks that both func and B.func exist, but then gives me an UndefVarError: func not defined.

This last one seems like a bug. As for all the previous examples, they all seem okay, but I can’t say that I have a good mental model for why exactly they behave the way they do. Is there one?

2 Likes
2

Nice resume, all that seems perfectly correct (and helpful).

I think the way to think about these things is about namespaces.

import A: func brings func to the namespace where that was executed. If that namespace is module B, then B.func is available. If inside module B you add export func now, func is available where using B was executed.

using A will put then func into the current namespace if it is exported in A (the same thing as above). If that was executed inside module B, now B.func exists.

import A does not bring A.func into the namespace, thus if that was executed inside module B, there will be no B.func.

(I don’t think you can import or export one method but not the other, of the same function. That is one of the reasons type piracy is highly discouraged).

Concerning the “bug”: actually the problem there is that Julia does not complain of a module exporting something that is not defined (it can’t really do that, because the module is parsed sequentially, thus it cannot know if there will be a definition later). A linter could possibly do that. But here the REPL does not seem to try to autocomplete, while B.<tab> does (with a stranger name that does not overlap with any other function that is clearer):

julia> module B
           export foobar
       end
Main.B

julia> foo # tab (nothing appears)

julia> B.foobar # does autocomplete
ERROR: UndefVarError: foobar not defined
Stacktrace:
 [1] getproperty(x::Module, f::Symbol)
   @ Base ./Base.jl:35
 [2] top-level scope
   @ REPL[2]:1
2 Likes
3

I started this post before @lmiq completed his post, so sorry for any overlap.

There’s an error because you’re exporting a function that doesn’t exist (A.func exists, but not func). So I would say that’s an error on the package developer’s part, not on Julia’s part.

But there will be B.A (and B.A.func by extension) because import A brings A into the namespace.

The way I think about it is that functions (and other global objects) belong to modules, while methods belong to functions. So adding a method to an existing function will not create a new function but will add to the function’s method table. So even in this setting:

one would get the methods that B defines for A.func automatically because B defines methods for A.func, not B.func (or some other function).

EDIT:

Just to be extra clear, the B.func that is accessible from the REPL is the same object as the func defined in A. Even though B is used to access the function, B does not have it’s own func, it just refers to A’s func.

EDIT 2:

I guess if you wanted B to have func with methods from A and B without adding methods to A.func you could do something like

module B
import A
func(args...) = A.func(args...)
func(i::Int) = i
end

In this case, A.func and B.func would be different objects, so adding methods to B.func would not add methods to A.func.

1 Like
4

It does if you add

julia> using .B
1 Like
5

Oh, yeah, I agree that this didn’t make much sense in the first place. I think I would have expected Julia to complain that I’m exporting a non-existing object, either while parsing B or if it can’t do that due to “sequential parsing” then when I’m importing B.

2 Likes
6

I have the impression that that problem falls into the same category as the usual complain that module dependencies are not parsed automatically. For instance, this errors:

julia> module A
           module B
               using ..C
           end
           module C
           end
       end
ERROR: UndefVarError: C not defined

one can ask: why can’t the the module be completely parsed before throwing the error? This has been exhaustively discussed in that “proper modules” thread, but I don’t remember the answer :stuck_out_tongue: .

1 Like
7

So it seems like it comes down to this:

I should think of the “object” canonically referred to as A.func basically as the methods table. This object is a singleton, in the sense that there can be an arbitrary number of references to it with different names (using A: func as foo) and in different namespaces (using A in module B → B.func), but they’re all pointing to the same object. The methods table gets filled with any method definition that Julia comes across under any name, anywhere. There is no way to to prevent methods from being added to the methods table: if the compiler can see them, they’ll be added. I’m glossing over some details like the “world age counter”.

Does that sound right?

The export statement has nothing do with any of this. It’s only meaning is to inject names into a namespace when importing.

Oh, and there’s some questionable behavior of Julia checking or not checking whether exported names actually exist, but that’s an unrelated discussion :wink:

2 Likes
8

As far as I understand, yes.

Hence

If everyone avoids type piracy, it probably doesn’t matter that a function gets methods added to it even in packages that are not directly used.

2 Likes
9

I think having access to all the methods of a function is an unavoidable design decision (even ignoring compiler’s stuff). Imaging I do import CoolArrays: CoolArray, should I be forced to explicitly import all the methods from the Iteration, Indexing, Abstract Arrays, etc “interfaces” into scope every time? Even if most of their functions (but not CoolArray specific methods) are already into base and are usually available? Should all developers be forced to re-export them? And that’s just one type.

1 Like