Reserving names in Base

In view of recent discussions on inner products https://github.com/JuliaLang/julia/issues/25565 and the move of basic stats functions https://github.com/JuliaLang/julia/pull/27375, I wanted to ask:

One of the big reasons for having things in Base is that packages can seamlessly coordinate: Packages A and B extend a base function and then can cooperate, without knowing about each other. If no suitable name / abstract interface exists in Base, then, well, as far as I understood there needs to be a fooBase package containing no actual code, only abstract types and function definitions. Then A and B both depend on fooBase and can cooperate.

Now, fooBase is really tiny (NPM left-pad style), and introduces only a tiny amount of overhead. So it might make sense to have some non-exported Base.Stubs.xyz as coordination points. That is: The bar for exported code in Base is much much higher than the bar for a non-exported empty function declaration. And I think that, in the example of inner or the recent PR for Banach-space norms https://github.com/JuliaLang/julia/pull/27401, much of the proponents good arguments for inclusion in Base could be met by giving them their reserved (non-exported, in a submodule!) name.

Indeed, I could imagine quite a few more useful name reservations, like e.g. distances (now in Distances.jl). In some sense, julia Base wants to be a kind of IANA (having a global DNS root is pretty useful for coordination, even though you of course want the option of locally resolved name-spaces).

What do you think?

1 Like

See https://github.com/JuliaLang/julia/issues/2327 and therein linked threads.

Thank you for re-raising the issue @foobar_lv2, it definitely deserves attention.

Widely used names such as inner need to be exported in an abstract fashion in order to let package writers generalize and collaborate more easily with each other. I agree with the shrinkage of Base in terms of implementations, but I extremely disagree with what is happening when it comes to “concepts” or “names” also being excluded.

Julia Base should have a single dummy file exporting widely used concepts:

export inner, norm, distance, ...

Some sort of math API that package writers could exploit to achieve generics in scientific code. An API that would evolve with the language as it matures.

This is the most natural solution to the problem, but as you can read in the issues linked above, core devs are resistant to make it happen. Their reasons are beyond my understanding.

Julia Base should have a single dummy file exporting widely used concepts:

There is no need for exporting them and polluting the common namespace; I would be quite opposed to that. Declaring is enough. Even having a Misc stdlib with submodules like Misc.geometry is enough for having a coordination point for package authors (that is not fooBase with the possibility of competing fooBases ).

Would it be a pollution? Aren’t these names standardized? I don’t see how it would interfere with package writers and users to have let’s say inner in global namespace.

To me, it seems like an approach like this is inherently non-scalable, would create an endless bikeshedding of what names should be included and what meaning they should have.

4 Likes

Good point, I think this can become a real issue if the math API has more than 10 names in it. I have very basic names in mind corresponding to very abstract mathematical spaces (topological, vector, normed, …). Things that are so standard that no one would debate semantics. In either case, I get the point.

Hope a solution is proposed that addresses this limitation. It is a big one in my opinion.

It is important to remember that the reason we share namespaces is so we can write generic code. In order to do so there needs to be some meaning to the generic function. For example, the documentation for the generic push! is

  push!(collection, items...) -> collection

  Insert one or more items at the end of collection.

So there needs to be some guidance from Base or by convention how these generic functions should be used. If a package extends topological what does this mean? The place where this has to be decided is where the generic function itself is defined, so some docstrings would need to go with e.g. distance. So how would you write the docstring for distance so you can write generic code for it?

2 Likes

My opinion of global names is that every programmer should (optimally) skim the manual for every function and type in his namespace. Hence everything non-essential in global namespace is pollution. Julia supports hierarchical namespaces just fine, and Base.Order.Reverse is a really good name: It has no way of ever colliding with anything or polluting tab-completion, and you can just write a using if this is too long for you to type.

Having something named inner that is not a dot-product makes sense in many contexts (inner html, inner expression, inner node, whatever).

I expressed myself poorly. When I said topological, I just had in mind general topological spaces. For these spaces, specific operations are well-defined that have a universal meaning. In that issue with inner products for example, this is an operation that is defined universally for all Hilbert spaces by definition. One could list a minimal math interface to describe these general mathematical spaces, that would put Julia apart from any other language I know (except Haskell which I think approaches this subject very seriously).

1 Like

I don’t think these use cases are relevant compared to the mathematical sense of the word. We are Julians, most of us are here for the scientific, mathematical side of things. HTML or the notion of “inside” versus “outside” is off-topic in my opinion.

Agree completely. This could evolve into a bunch of conditional code on various julia versions for figuring out where to put methods, and everyone fighting to get function names added to base to make them more convenient to use.

An alternative way to deal with this are macros to make method-merging easier (and independent of the ordering, changes in base, etc.). Basically, along the lines of GitHub - chakravala/ForceImport.jl: Macro that force imports conflicting methods in modules and making Is merging method definitions possible? · Issue #1 · chakravala/ForceImport.jl · GitHub more usable. Then you can basically write code which says “extend whichever method is already there” and the order of package includes would be irrelevant.

Not every meaning of “inner” needs to be consistent with a mathematical inner product, especially if julia intends to be a general purpose language or at least interface with them. You don’t want dispatch clashes, but if the meanings are truly distinct, then they would have no overlap of supertypes for dispatch. Any overlap is a clear signal that the “pun” is a bad idea.

That is what I am still trying to grasp. The identity of the language is not clear to me at all. Sometimes I hear people saying it is a language for technical/scientific computing. Sometimes I hear it is general. I like the former goal, and I started using the language because of that premise. If the premises changed, it would be great to have an official statement from the devs so that I can readjust my future plans.

I see Julia as a general purpose language with a focus on scientific/technical computing.

6 Likes

This is not necessary to determine. Even if Julia is focused on scientific computing, there is still no reason to formalize every single use of inner as a fully-consistent set of inner products operating on the same set of types. Even if that works with inner, it becomes less and less feasible with other function names as the number of packages and the standard library increases (hopefully). A properly written generic programming language is perfectly able to handle concurrent “meanings” of a single function name… I would say it is almost a crucial test of the design of the language.

Agree, I am just saying that the scientific computing experience shouldn’t be compromised by generality. We all want generics, which is different from general-purpose.

The notion that a language for numerical computing cannot be general purpose is old, anti-Julian thinking. If there is a foundational principle of Julia it is that we set out to design a language where numerical things would be less special than they are in other languages and that integers and floats and arrays of them and all the basic numerical operations on them are just normal types and functions instead of needing to be special and built in.

8 Likes

Understand. Good discussing it, makes things clearer.

It turns out that what you need is roughly this:

  1. The ability to define lightweight immutable types, e.g. Int, Float64, Complex{Float64}, Rational{Int}.
  2. Parametric collections, e.g. Array{Float64} which can efficiently store lightweight immutable values inline.
  3. Generic functions with dynamic dispatch on all arguments that is fast enough to define things like + on simple types like the above lightweight immutables.
4 Likes