# What if a package consisted of multiple top-level files?

**URL:** <https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465>\
**Category:** General Usage\
**Created:** [December 2, 2017, 11:45pm UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465 "2017-12-02T23:45:19Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [December 2, 2017, 11:45pm UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/1 "2017-12-02T23:45:19Z")

</div>

It seems to me that it might make sense to have a package consist of multiple (dozens?) related files that the user does not necessarily need all at once. What is the standard mechanism for dealing with this? (Is there one?)

So, let’s say the package is called `P1`. There is a `P1.jl` file (top-level), but then there are let’s say three other files. When as a user I want to use some function I may need one of these three. Presumably there is a module in that file. How do I load that module for `import`? Do I have to use `Pkg.dir()`?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [December 3, 2017, 12:24am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/2 "2017-12-03T00:24:22Z")

</div>

> [@PetrKryslUCSD](#):
>
> What is the standard mechanism for dealing with this? (Is there one?)

Namespaces, i.e. using inner modules.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [December 3, 2017, 12:32am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/3 "2017-12-03T00:32:50Z")

</div>

Inner modules would require though that the entire file that defines the enclosing module will be loaded, correct? That is something I want to avoid.

We are all familiar with packages that takes a long time to load. I want to make it possible to construct lean-and-mean subsets of the package.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [December 3, 2017, 12:48am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/4 "2017-12-03T00:48:40Z")

</div>

> [@PetrKryslUCSD](#):
>
> Inner modules would require though that the entire file that defines the enclosing module will be loaded, correct? That is something I want to avoid.

Specifying what is getting loaded up front allows for precompilation. This will likely be a much larger performance increase than trying to “hot load” new code. For example, DataFrames loads ~10x faster with precompilation than without.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [December 3, 2017, 12:49am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/5 "2017-12-03T00:49:59Z")

</div>

So are you saying that no matter how big the package gets pre-compilation will take care of the load time?  
Isn’t there a price that one has to pay in terms of the resources consumed that are involved in loading a huge package?

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [December 3, 2017, 12:58am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/6 "2017-12-03T00:58:07Z")

</div>

This question seems to tie in with the question about package sizes:

> [@Package size and scope?](https://discourse.julialang.org/t/package-size-and-scope/7309/):
>
> Hi All – Recently started developing packages. Was wondering how everyone feels about package size and scope. Do you feel like there is a minimum or maximum size/scope a package should be? As a Developer: Do you include just a few things in a package (and do them well)? Or do you put in everything but the kitchen sink? As a User: Do you include a package with 10 functions to just use 1 of them? Or do you not want to be bothered with a la carte packages? For context, (and a tad less b…

Maybe what you want is for some package to be split into smaller packages, to simplify it

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [December 3, 2017, 1:03am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/7 "2017-12-03T01:03:43Z")

</div>

Yes, indeed. But at the same time I want to keep the relationship of the “sub”-packages intact. Right now I have the package split up into dozens of modules. These however get all loaded at the same time whenever I say `using MyPackage`, whether I need them or not.

So really what I’m looking for is some way of having multiple packages under the umbrella of one top package. Does anyone know if there is a beast like this around already?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [December 3, 2017, 1:08am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/8 "2017-12-03T01:08:58Z")

</div>

> [@PetrKryslUCSD](#):
>
> So are you saying that no matter how big the package gets pre-compilation will take care of the load time?
> 
> Isn’t there a price that one has to pay in terms of the resources consumed that are involved in loading a huge package?

Julia is JIT compiled and the cost of compiling functions is typically much larger than just “loading” the code. It is unlikely that you will beat the 10-50x performance boost of precompilation, while keeping the same functionality.

Your package [GitHub - PetrKryslUCSD/FinEtools.jl: Finite Element tools in Julia](https://github.com/PetrKryslUCSD/FinEtools.jl) is currently running with precompilation off. Enabling it makes load time go from 5 seconds to 0.08 seconds. What part of those 0.08 seconds are you trying to improve and do you think it is relevant comparing to other things like compilation time when functions are actually called?

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [December 3, 2017, 1:11am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/9 "2017-12-03T01:11:30Z")

</div>

> [@PetrKryslUCSD](#):
>
> So really what I’m looking for is some way of having multiple packages under the umbrella of one top package. Does anyone know if there is a beast like this around already?

Have you tried something like this?

```Julia
module stuff
    module here
        export x
        x = 1
    end
end

```

then you can do `using stuff` without it loading the `here` module. Then if you need to load that too, you can execute `using stuff.here` and then that will be loaded. Is that what you are want?

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [December 3, 2017, 1:31am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/10 "2017-12-03T01:31:39Z")

</div>

Maybe I’m not understanding, but this is what I get:

```julia
julia> module stuff
           module here
               export x
               x = 1
           end
       end
stuff

julia> using stuff

julia> stuff.
eval here
julia> stuff.here
stuff.here

julia> stuff.here.x
1

```

I. e. no need to do `using stuff.here`. It is already there when I do `using stuff`.

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [December 3, 2017, 1:35am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/11 "2017-12-03T01:35:52Z")

</div>

This is what I get:

```nohighlight
julia> module stuff
           module here
               export x
               x = 1
           end
       end
stuff

julia> using stuff

julia> x
ERROR: UndefVarError: x not defined
Stacktrace:
 [1] macro expansion at ./REPL.jl:97 [inlined]
 [2] (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:73

julia> using stuff.here

julia> x
1

```

So you have to call `using stuff.here` in order for the `here` module to be loaded into `main`.

Of course, you can always do `stuff.here.x` no matter what, but that is independent of `using`.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [December 3, 2017, 1:52am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/12 "2017-12-03T01:52:40Z")

</div>

But if you can do `stuff.here.x` then presumably that code is already loaded?

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [December 3, 2017, 1:58am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/13 "2017-12-03T01:58:29Z")

</div>

Oh okay, well if you want to have code in your package that doesn’t get loaded, then you will have to just simply include it in a separate file like `src/extra.jl`. Then you would have to do `cd(Pkg.dir("Name"))` and then `include("src/extra.jl")` to load that code.

However, you’re probably better off using precompile.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [December 3, 2017, 2:02am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/14 "2017-12-03T02:02:50Z")

</div>

That is a good point. However my package has around five thousand lines, give or take. What if I managed to rewrite Nastran with its five million lines of code? Are we then talking about eighty seconds to load the package?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [December 3, 2017, 2:13am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/15 "2017-12-03T02:13:11Z")

</div>

No, then you would use something like [https://github.com/JuliaComputing/static-julia](https://github.com/JuliaComputing/static-julia) to AoT compile your code into a binary.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [December 3, 2017, 2:18am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/16 "2017-12-03T02:18:11Z")

</div>

That is a good idea. Thanks!

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [December 3, 2017, 2:42am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/17 "2017-12-03T02:42:22Z")

</div>

Another thing you could do is put your extra module `ExtraModule` into `mod/ExtraModule.jl` and then in your main module `PkgName.jl` script add a line like this

```nohighlight
push!(LOAD_PATH, joinpath(Pkg.dir("PkgName","mod")

```

Then you will be able to do `using ExtraModule` and it will find the code for it in that folder.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [December 3, 2017, 2:43am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/18 "2017-12-03T02:43:10Z")

</div>

That is true. Neat! Thanks.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [December 3, 2017, 3:09am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/19 "2017-12-03T03:09:44Z")

</div>

`Pkg.dir` is not ideal to use like this because it will fail if the package is installed somewhere else than `Pkg.dir`. Use relative includes instead like `joinpath(@ __DIR__ , "..", "mod")`.

Also, relying on user controlled global behavior like `LOAD_PATH` is quite fragile and possibly confusing for users.

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [December 3, 2017, 3:25am UTC](https://discourse.julialang.org/t/what-if-a-package-consisted-of-multiple-top-level-files/7465/20 "2017-12-03T03:25:27Z")

</div>

True, at that point it probably just makes sense to split it into multiple separate packages anyway, since you’d have to worry about `ExtraModule` possibly conflicting with another package name, the way it is used at that point is essentially as if it is another package anyway.