Organization of multiple modules in same package


#1

I’m developing a package which defines a number of types. Some types are helpers and some are specific implementations (parametric or abstract) of other types from this package.

I’ve been poking around popular packages to see how they might handle this. I’ve seen different styles. I’m all about extensible patterns, so if it’s not extensible gracefully I’m a little annoyed.

The most common package looks like this: One module defining more general/generic type(s) or interface(s) and including more specific code via direct file inclusion. The “module” file exports stuff for the included files.

module Foos

export 
Foo,
bar
    
struct Foo{T}
  x::T
end

include("bar.jl") 

foo(x::Foo{T}) where T =  x.x+x.x

end

bar.jl

foo(x::Foo{String}) =  x.x*x.x

bar() = "hey"

This is really just breaking a large Foos.jl file into multiple files. Evidence of that is that Foos.jl exports bar.jl functions/types.

To extend this pattern to include more types/interfaces in modules I’d need one package per and compose via using. If they are very small types/interfaces, I don’t know that I really want to break them up into multiple packages.

So I’ve seen this kind of pattern for multiple modules in a single package.

Foos.jl

module Foos
include("Bars.jl")
using .Bars

Problem here is I can only do that for Bars once inside Foos tree of includes. So if Foos includes Bars and Bazs, but Bars and Bazs want to both use a module Bats I get an error. This isn’t very extensible.

In my specific case Bats should really be it’s own package. However, I could see other cases where Bats was a utility module that only made sense in the Foos context. Where it should probably not be a module I suppose and just included and I trade an error for some warnings when Bats functions etc are redefined.

Seems I’m limited to linear or flat include/using relationships between files and modules in one package. Breaking packages up anytime I need to go beyond this seems the only sensible way. If I want to start off extensible that means I should always start a new package when I have a new type/interface. This seems impractical.

Any suggestions for more extensible patterns for large projects where I can have multiple modules in one package?


#2

Perhaps I am misunderstanding, but I don’t see defining dozens of modules within a package as a problem (FinEtools).
See FinEtools.jl, which groups multiple modules under one umbrella.


#3

I didn’t intend to say it’s a problem, just a not known solution to me and a quick peek around didn’t satisfy me. Julia provides lots of building blocks and it’s still young, so I didn’t doubt there were good solutions possible.

FinEtools organization looks great. I was messing around a bit and started converging on something quite close to what you have there. The top level FinEtools.jl files is serving as a kind of project file. I have some questions still, but I’ll study this first.

Thanks!

A trap I fell into here was treating modules like C++ classes/files. You’d usually organize your include pattern top down in C++, but with the modules it appears you actually want to have the modules used by other modules to appear first. Otherwise you get LoadErrors… oh yeah, isn’t this what include guards are there for in C++? Wonder if something similar would be worth doing in julia.


#4

Maybe this could be of use https://github.com/simonster/Reexport.jl?


#5

Alright I messed around with the idea of emulating the C/C++ convention.

In C/C++ you would #include everything in a tree-like manner. Which is nice because it typically would mirror how the files are laid out on disk, and if you are doing OOP mirrors your inheritance relationships. Then you use a dirty little preprocessor macro to guard against including a file twice. Since all those nested includes are really just cutting a pasting into a top level file.

Here’s how you would do it in Julia.
I emulate the application or library environment with a top level module. I wrote a dirty little macro to guard against adding a module twice. Here’s an application built out of 4(+1 ) modules with the Ds module used twice at the “leaf” level.

It would be 5 files, but I did the job of include(“Xs.jl”) cut-and-paste for brevity/clarity.
Note: #include goes before your namespace declarations.
so in julia Bs.jl would look like this

include("Ds.jl")
@once module Bs
....

Application.jl

module Application
  macro once(exp)
    if exp.head == :module
      if !isdefined(__module__, exp.args[2])
        esc(Expr(:toplevel,exp))
      end
    end
  end

  # include("As.jl")
  @once module As
    export A, foo
    abstract type A end
    foo(x::A) = 0
  end
  # end As.jl

  # include("Bs.jl")
  @once module Ds
    struct D
      d::Float64
    end
    bar(x::D) = x.d^2
  end

  @once module Bs
    using ..As: A
    using ..Ds: D
    import ..As.foo
    struct B <: A
      b::Float64
    end
    foo( x::D, y::B ) = x.d*y.b
  end
  # end Bs.jl

  # include("Ds.jl)
  @once module Ds
    struct D
      d::Float64
    end
    bar(x::D) = x.d^2
  end

  @once module Cs
    import ..As.foo
    using ..Ds: D
    foo( x::D ) = x.d
  end
  #end Ds.jl
end

julia> include("App.jl")

julia> Application.As.foo( Application.Ds.D(2.0), Application.Bs.B(10.0))
20.0

I haven’t sorted out exporting, I’ll check out reexport. Thanks for the link!
I’d also like to write using As instead of using …As inside the sub-modules, but I’m ok with that. It’s a bit like #include “As.hpp” instead of #include <As.hpp>


#6

I edited my macro to return an Expr rather than eval directly. Not exactly sure why :toplevel is needed, but it is.