Import types "above" a module with ..: for consistency and brevity?

General Usage
1

You can import stuff from “above” with ..Thing, such as:

module Mod

# Define some common types.
struct T1; field::Int64; end
struct T2; field::Float64; end

# Define a sub module. Maybe this would even live in a different file. It assumes that
# T1 and T2 are available in its parent, but it doesn't need to know who its parent is.
module Sub
    import ..T1, ..T2
    f(x::T1) = T2(2 * x.field)
end

end

That works fine. But once you start importing a bunch of stuff, it would be really nice if you could change this:

    import ..T1, ..T2

to

    import ..: T1, T2

which would be consistent with importing from a module with a given name, like so:

    import ...Mod1: T1, T2

The problem with this final line is that it requires Sub to know who its parent is (Mod1). If Sub is intended to be used by a lot of different utilities, then it naturally can’t know who its parent is.

So I think ..: is nice, clean, and consistent. Does this seem like a good idea, or am I missing something?

4 Likes
2

I get the consistency argument, but I’m not sure I understand this one:

I might have misunderstood something, but I think that in any case, Sub has to know a lot of things about the module from which it imports things: at least it has to be sure that it defines types T1 and T2, and these types have to follow an specific interface. And how would Sub know all that, but not the name of Mod1 itself?

In other words, I think that putting a line such as import B: foo in module A defines some form of dependency of module A to module B (and if they were packages, this would be enforced by the packaging system). But how can A depend on something it does not know?

In short, I agree with you about the syntax for the sake of consistency, but I would be interested in a (maybe less contrived) use case to motivate it in practice.

3

Hi @ffevotte. Good question. Here’s a simple example that captures the essence of a real problem we’re having at my company: Suppose we have a MyTypes module that exports a bunch of common types. Those types are on the interface for functions in the Subsystem module. Now we want to build two separate systems that use Subsystem. Here’s an example. Suppose the modules are included from separate files (they’re just written this way so there’s a single working example file).

Here’s a system that needs to use the functionality of Subsystem directly.

module SystemA

    # Included from MyTypes.jl.
    module MyTypes
        export Type1, Type2
        struct Type1; field::Int64; end
        struct Type2; field::Float64; end
    end

    using .MyTypes

    # Included from Subsystem.jl.
    module Subsystem
        export f
        import ..MyTypes: Type1, Type2 # This line breaks modularity!
        # import ..SystemA: Type1, Type2
        f(a::Type1, b::Type2) = Type2(a.field + b.field)
    end

    using .Subsystem

    # Now for whatever this module was supposed to do.
    x = f(Type1(1), Type2(2.0))

end

Here’s a different system, one that needs to use the functionality of Subsystem indirectly. [Edit: This is intended to be totally separate from SystemA, like a different program.]

module SystemB

    # Included from MyTypes.jl.
    module MyTypes
        export Type1, Type2
        struct Type1; field::Int64; end
        struct Type2; field::Float64; end
    end

    using .MyTypes

    # Included from SomeOtherSubsystem.jl.
    module SomeOtherSubsystem

        export g
        import ..MyTypes: Type1, Type2
        # import ..SystemA: Type1, Type2

        # Included from Subsystem.jl.
        module Subsystem
            export f
            import ..MyTypes: Type1, Type2 # This line breaks modularity!
            # import ..SystemA: Type1, Type2
            f(a::Type1, b::Type2) = Type2(a.field + b.field)
        end

        using .Subsystem

        g(a, b) = f(Type1(b.field), Type2(a.field))

    end

    using .SomeOtherSubsystem

    # Now for whatever this module was supposed to do.
    x = g(Type1(1), Type2(2.0))

end

SystemA.x
SystemB.x

This doesn’t work. In order for Subsystem to work in the second case, its import statement would need an extra period, because it’s deeper here. Hence no single Subsystem.jl file works in both cases.

If the import ..: Type1, Type2 syntax worked, then this type of modularity would work.

This is all part of a bigger picture that looks like this: My company needs all of this code in a single repo. We want modularity. Solutions:

  • If we break things into modules like this, modules actually can’t be used in multiple contexts, so this doesn’t work.
  • If we break things into local packages and ]dev them, then there are new problems with modularity ["Unsatisfiable requirements" when local packages use local packages (fixed in v1.4)].
  • If we create a local registry for our own package, that’s also problematic [ibid].
  • This only working solution we’ve found is to avoid making modules and packages and just make a bunch of .jl files with functions and types in them, and then include all those files in one big module (possibly Main). This obviously has its own problems, like name conflicts and the fact that nothing can be precompiled, etc., so the startup time is now really slow.

That’s all just context to answer your question, François. This thread should still be about whether or not ..: is useful and should be prioritized.

4

Well, I’m sorry to derail this further but, on the other hand, I feel like this proposal is more likely to be prioritized if people understand why it is so useful to you.

One problem I see with the way you do things, is that the same code is included in several different contexts, creating types (Type1 and Type2) and functions (f) that should be the same, but actually are not. To be clear: in what you describe above, SystemA.MyTypes.Type1 and SystemB.MyTypes.Type1 are two different types that happen to have the same structure. I would say that this could cause unexpected behavior later on, especially if these types are supposed to be part of a common interface.

Here is how I would structure things:

# Only one MyTypes module
module MyTypes
  export Type1
  struct Type1; field::Int; end
end

# Only one SubSystem module
module SubSystem
  using ..MyTypes
  export foo
  foo(x::Type1) = x.field
end

# Module SystemA uses the top-level MyType module
module SystemA
  using ..MyTypes
  using ..SubSystem
  test() = foo(Type1(1))
end

module SystemB

  # This makes MyTypes and SubSystem available in the current namespace
  # ...and therefore also in submodules, using the .. syntax
  import ..MyTypes
  import ..SubSystem

  # Submodules of SystemB can also use the same top-level MyType module
  # without having to know about their depth in the module hierarchy
  module SomeOtherSubSystem
    using ..MyTypes
    using ..SubSystem
    export bar
    bar(x::Type1) = 2*foo(x)
  end

  using ..MyTypes
  using .SomeOtherSubSystem
  test() = bar(Type1(1))
end

SystemA.test() # --> 1
SystemB.test() # --> 2

Now, I would even go one step further and imagine that everything is in its own package. In a real application, the code base should reside in a package, that defines a top-level module of the same name. When the package environment is activated, this top-level module can be referred to in absolute syntax (without leading dots) from anywhere. Therefore, submodules can also be accessed to using an absolute path starting from the top-level module.

The following example is designed to run in the REPL, where everything is defined in the Main module. If you want to transpose it to a project, simply replace every occurrence of Main with the name of the top-level module:

# Only one MyTypes module
module MyTypes
  export Type1
  struct Type1; field::Int; end
end

module SubSystem
  # Refer to MyTypes using an absolute path starting with the top-level module
  # (replace Main with the name of your top-level module, i.e. the name of the package)
  using Main.MyTypes
  export foo
  foo(x::Type1) = x.field
end

module SystemA
  using Main.MyTypes
  using Main.SubSystem
  test() = foo(Type1(1))
end

module SystemB
  module SomeOtherSubSystem
    # No need to know how deep we are, since we now use absolute module paths
    using Main.MyTypes
    using Main.SubSystem
    export bar
    bar(x::Type1) = 2*foo(x)
  end

  using Main.MyTypes
  using .SomeOtherSubSystem
  test() = bar(Type1(1))
end

SystemA.test() # --> 1
SystemB.test() # --> 2

Again, this has almost nothing to do with your proposed syntax and I’m sorry for the thread derailment, but would such a structure make sense in your project?

2 Likes
5

Wow, @ffevotte, thank you for that amazingly detailed reply!

Let me clarify two things:

  1. SystemA and SystemB are intended to be totally separate programs. I understand that the two instantiations of MyTypes are different there. That’s intended. The two systems just happen to be next to each other in the example above. Maybe I should have used two separate code blocks. I’ve edited my post to try to make that a little clearer.

  2. Your proposal is essentially to put all possibly-common modules at the top level, and then import their functionality for each level going down. What I don’t like about this is the lack of modularity. Somehow, the top level needs to know what to import, so it needs to know that something inside of SystemB uses Subsystem.

  3. Also, instead of having types/function “cascade” down each level, you suggested importing them directly from the named top level, which I like. However, the only way things can imported from the “top level” is if they know the name of the top level. If SystemA and SystemB are in two separate packages, then this still doesn’t work.

  4. I would like to use packages. What’s great about this of course is that any module can say, e.g., using MyTypes, and everyone gets the same types. There’s no necessary “routing” of types down a level, and down a lever, etc., to the intended user. Nor does the top-level module need to know about everything that everyone might possibly use and load it at the top level. It’s sort of automatically loaded at some kind of top level. BUT! I need everything in the same repo. It’s the only way my code base makes sense (there are a bunch of reasons for this that have nothing to do with Julia). That leaves me trying to use unregistered local packages via ]dev path/to/LocalPackage. The problem with this approach is that deving local packages doesn’t really work as a sustainable workflow (see my “Unsatisfiable requirements” link above).

Maybe the best way forward is some kind of first-class support for local, unregistered packages. I’ll make a new post about that, and we’ll see where it goes.

In the meantime, I’m going forward with both local modules (including them, and cascading their types/functions down and down and down) and local, unregistered packages (via ‘dev path/to/Package’) to see which is the least painful. Though the least painful thing so far is to simply do away with the idea of little modules and just make a bunch of functions and types in separate files, and then include them in either the module for SystemA or SystemB. This is more like mixins, and it’s hard too because the order of the includes has to be just so.

6

Here’s a link to the new post about first-class support for local, unregistered packages.

7
8

That’s cool. Thanks @Tamas_Papp. Still a work in progress, I see, but it’s good to know about.

I’ll have to think about how this would possibly work in the context of private repos. E.g., our sim’s going to run in Docker without Git credentials, so it won’t be able to hit our private repo, but as long as we already have it checked out somewhere, then… maybe something like this would work?