Why is Module concrete?

I repeatedly find myself thinking that having a unique type for every module would open possibilities, like dispatching and use of Modules as type parameters.

There are workarounds, e.g.:

https://github.com/SciML/RuntimeGeneratedFunctions.jl/blob/cee1c8a5fa462eacc18cb765d789f6f87d602543/src/RuntimeGeneratedFunctions.jl#L82

So it seems that there is no fundamental cause for this, but I admit I do not know anything about the implementation of modules.

2 Likes

Just as a vocabulary thing, the Module type is a concrete leaf-type — that simply means that it is not abstract. It’s not a bitstype — that’s what is required for something to be used as a type parameter.

julia> isconcretetype(Module)
true

julia> isbitstype(Module)
false

So I guess there are two distinct questions here:

  • Why can’t you use modules as type parameters? That is, why aren’t they bitstypes?
  • Why isn’t typeof(Base) different from typeof(Core) — with a type like Module{Base} or some such?
5 Likes

Maybe I was not clear enough: My problem is that there is only a single Module type, which is concrete, so it is impossible to subtype it.

When it comes to type parameters, I would use the subtypes - concrete module types - as type parameters, not Module itself.

So by analogy — there’s only one Int64 type. It’s a concrete leaftype that’s not subtype-able. But it’s a bitstype whose values are able to be used as a type parameter. This is how, e.g., an Array{Any, 1} is different from an Array{Any, 2}. Both 1 and 2 have the same type, but they’re able to be used as distinct type parameters.

Base and Core are both values of type Module like 1 and 2 are values of type Int64.

This is why I split your question into two parts :slight_smile:

5 Likes

Why not dispatch on ::Val{:ModuleName} ? You can get it always by nameof(ModuleName).

Seems like an interesting workaround, and nameof() itself is fast, but:

julia> @btime Val{x}() setup=(x=nameof(Core))
  3.153 ÎĽs (0 allocations: 0 bytes)

Also got similar results with struct TypedModule{TName} end. This seems like another question again…

There’s another problem with nameof — and that’s that it isn’t unique:

julia> module Algebra
           module Internals
           end
       end
Main.Algebra

julia> module SocialStudies
           module Internals
           end
       end
Main.SocialStudies

julia> nameof(Algebra.Internals)
:Internals

julia> nameof(SocialStudies.Internals)
:Internals

Now, as to why Val{x}() is expensive, this really gets to the crux of what I put as your “second question”. In short, Val{x} and types like it put values into the type domain. This is an inherently type unstable operation unless the value itself is a constant. Now, depending upon the downstream operations, you can sometimes recoup the cost. This is likely one aspect to the answer about why typeof(Base) == typeof(Core):

4 Likes

Thanks for the clarifications!

I am still curious and hope that someone knows the full answer!

I don’t think there is necessarily anything wrong with modules being their own types (like generic functions), but it also isn’t quite clear to me what that would accomplish exactly.

3 Likes

I beleive I remember @oxinabox showing an example where it would be quite nice if Modules were their own type, but I can’t remember what it was anymore.

1 Like

I have made a few examples of this.
For different reasons.
Many of which are breaking, or at least go with changing some other semantics.

When i was messing around with attaching citation info to things,
One way it can be done is to introduce functions like

citation(::typeof(foo)) = "2018, oxinabox, What if we attached things to things"

can’t do that for modules
Can do that for for module.eval though, which is a neat hack if you want to attach things to things in this trait-ish way.

It would be really nice if all exceptions had a type parameter that was the module that through them.
Then we could more easily do appropriate rethrowing,
via err isa Exception{<:SomeModuleICanSolve} || rethrow,
and deal with badly over used exception types like ErrorException and ArgumentException, since we would at least know what module they are from.

In general all namespacing could be replace by an implict module typed first argument that is magically inserted if not given.
Then ratehr than Foo.bar(x,y) one would have (but sugared away mostly) bar(Foo, x, y).
Which would be kind of elegant.
(Would definately want more clear sugar then i am outlining here.)

7 Likes

So here’s are some attempts at superficial non-expert non-definitive answers:

As far as why Modules aren’t isbits: they don’t expose any fields*. isbits objects are identified purely by their fields, which allows the Julia compiler to treat objects of the same type with identical fields as being identical. But with no (exposed) fields, I think the compiler might naively do some silly things when handling multiple modules. Of course, I imagine that’d be simple to address, but this implementation is the obvious, simple way to get them to behave the way we want in Julia itself.

And a superficial reason for them all being the same type is, well, it’s the obvious, simple way for them to work.

Only one of these two things would need to change for them to be used as type parameters.

* asterisk regarding fields of a module Modules are one of the very few data types that aren't actually defined in Julia itself; they do have fields but we don't expose them to Julia**.
** double-asterisk regarding exposing fields of a module

Ok, that’s only true in Julia 1.6+. But on earlier Julias they were bugged and you couldn’t access their values anyways.

julia> fieldnames(typeof(Base))
()
2 Likes

Here is my motivation:

I have created a plugin system that helps to break large software into smaller chunks, just like modules do, but with extras. Plugins are types, and the first argument of every function they provide to the outside world is the plugin itself, similar to how @oxinabox described, minus the sugar. (simple example)
This opens up some possibilities, like glueing and inlining multiple implementations of the same callback from different plugins, replacing a plugin on the fly, or even extending the lifecycle of plugins with staging.
Conceptually, plugins are still modules in my mind. I would like to convert them to real modules to reflect this, but without modules as type parameters it seems futile (even if possible e.g. with parametrizing on every callback separately).

OK, I see that this is not very convincing. :slight_smile: But the examples of @oxinabox also show that it would open up possibilities.

I also think that separating interface and implementation modules could have benefits (seems like the XXXInterfaces.jl naming convention starts to spread, showing the need for this), and that there is a natural hierarchy of modules. Both seem to come for free with typed modules.

I sometimes also dream of loading code based on types, not symbols, but I don’t really know where science fiction begins…

1 Like