# Loading packages programmatically in a different module

**URL:** <https://discourse.julialang.org/t/loading-packages-programmatically-in-a-different-module/52777>\
**Category:** General Usage\
**Tags:** question\
**Created:** [January 3, 2021, 4:10pm UTC](https://discourse.julialang.org/t/loading-packages-programmatically-in-a-different-module/52777 "2021-01-03T16:10:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![felix](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/felix/32/604_2.png) [@felix](https://discourse.julialang.org/u/felix)\
**Post date:** [January 3, 2021, 4:10pm UTC](https://discourse.julialang.org/t/loading-packages-programmatically-in-a-different-module/52777/1 "2021-01-03T16:10:46Z")

</div>

Hello, for an experimental idea I would like to try to create and manipulate modules on the fly, including letting them use packages. I’m struggling to understand/get hints from the docs about this, so I wanted to ask for any pointers on the following:

1. how do you use the `Pkg` machinery to calculate a set of packages given the deps and compat of a Project.toml file?
2. how do you load a given package in a foreign module?

Here’s my aspirational pseudo-code marked with `# ???`, grateful for any corrections or suggestions:

```julia
# the set-up
Mname = :Module1
deps = Dict{String,Any}(
    "CSV" => "336ed68f-0bac-5ca0-87d4-7b16caf5d00b",
    "DataStructures" => "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8")
compat = Dict{String,Any}(
    "julia" => "1.5",
    "CSV" => "0.8.2",
    "DataStructures" => "0.18.8")
filepath = "includethis.jl" # which does `using CSV, DataStructures`

# the action
M = Module(Mname)
packages = resolve(deps,compat) # ???
make_available(M, packages) # ???
Core.include(M, filepath)

```

Happy new year!

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [January 4, 2021, 1:20pm UTC](https://discourse.julialang.org/t/loading-packages-programmatically-in-a-different-module/52777/2 "2021-01-04T13:20:55Z")

</div>

The package interface is meant to be declarative, so this is not part of the exposed API. That said, you can call the [resolver](https://github.com/JuliaLang/Pkg.jl/blob/master/src/Resolve/Resolve.jl), and then load with `eval` and `using`.

I am not sure what the purpose is, but this may not be a good idea though.

---

<div class="post-metadata">

**Author:** ![felix](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/felix/32/604_2.png) [@felix](https://discourse.julialang.org/u/felix)\
**Post date:** [January 4, 2021, 7:28pm UTC](https://discourse.julialang.org/t/loading-packages-programmatically-in-a-different-module/52777/3 "2021-01-04T19:28:01Z")

</div>

Thanks, that may be helpful for the resolution part, but the problem remains of making the (right versions of) packages available to the module `M`.

I looked through Resolve, but couldn’t follow the flow of it from reading. (E.g. none of the functions in Resolve.jl take a dictionary of dependencies or compatibilities, they all want a graph or a solution vector from somewhere else.)  
I guess from API.jl that the key resolution action happens in the following ([link](https://github.com/JuliaLang/Pkg.jl/blob/0c3ccb119efd8b3182c114599abe78bc4f89dd4c/src/API.jl#L174)), which acts on a `ctx::Context` object:

```julia
    project_deps_resolve!(ctx, pkgs)
    registry_resolve!(ctx, pkgs)
    stdlib_resolve!(pkgs)
    ensure_resolved(ctx, pkgs, registry=true)

```

The `Context` object has an `EnvCache` inside it.

**The key thing is that** , when I load the dependencies `X,Y,Z` inside the module `M` by using calls to `using X,Y,Z` in the file `includethis.jl` which is executed thanks to the call `Core.include(M,"includethis.jl")`, **I want `M` to access the packages and versions specified through `deps` and `compat` specifically for `M`, and not just fall back on whatever globally available packages there are**. Is that controlled by the currently relevant environment/`EnvCache`? How do I change the “environment” that `M` operates in?

Do I have to use something like `Pkg.activate`? All the machinery I have seen seems quite intimately linked to having a particular folder, containing a particular set of files (`Project.toml` and `Manifest.toml`), but I can’t tell if all of those big guns are necessary for what I want? I don’t have a folder with those files, so I would have to create a temporary set-up just for that, and even then I still don’t know how to associate it to `M`.

---

<div class="post-metadata">

**Author:** ![MA\_Laforge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ma_laforge/32/385_2.png) [@MA\_Laforge](https://discourse.julialang.org/u/MA_Laforge)\
**Post date:** [January 4, 2021, 10:04pm UTC](https://discourse.julialang.org/t/loading-packages-programmatically-in-a-different-module/52777/4 "2021-01-04T22:04:20Z")

</div>

> [@felix](#):
>
> **The key thing is that** , when I load the dependencies `X,Y,Z` inside the module `M` by using calls to `using X,Y,Z` in the file `includethis.jl` which is executed thanks to the call `Core.include(M,"includethis.jl")` , **I want `M` to access the packages and versions specified through `deps` and `compat` specifically for `M` , and not just fall back on whatever globally available packages there are**. Is that controlled by the currently relevant environment/ `EnvCache` ? How do I change the “environment” that `M` operates in?

Hard for me to tell if you are trying to create something special/new for Julia or if you simply don’t now that these things already exist.

# Assuming I understand your issue correctly

### How to correctly create “software modules”

When you create a new package (basically a “software module”), you should have the following structure:

```julia-auto
Package1 #Base directory of your "software module"
├── Project.toml #(required, esp. for what YOU want)
├── Manifest.toml #(optional)
├── src
│ ├── Package1.jl
│ ├── P1SubFile1.jl #If package too big for 1 file (typical).
│ └── P1SubFile2.jl
... #Typically want a "test/" directory, but we'll ignore that

```

Note that I chose the name “`Package1`” instead of “`Module1`” to highlight the fact that “software modules” are packages (and Julia "`module`"s are namespaces).

Also note that you can generate the skeleton structure using:

```julia-auto
julia> ]
pkg> generate Package1

```

IMPORTANT:

- Don’t forget that the code from Package1.jl file must be encased in a “`module Package1` … `end`” block to function properly.

### Making a package available

Once this package is generated, you must make it available to your current project (or current package) environment with the help of `pkg> add` or `pkg> dev`:

```julia-auto
julia> ]
pkg> dev /abs/path/to/Package1

```

I chose `dev` over `add` because, at this stage, you probably don’t want to `git commit .; pkg> up Package1` every time you make a change in `Package1`.

### Controlling dependencies

> [@felix](#):
>
> **I want `M` to access the packages and versions specified through `deps` and `compat` specifically for `M` , and not just fall back on whatever globally available packages there are**.

That’s what the `Package1/Project.toml` file is for. To add dependencies to your package you must first activate this package (which then allows you to edit its `Project.toml` file):

```julia-auto
julia> ]
pkg> activate /abs/or/relative/path/to/Package1

```

then you can to add whatever packages `Package1` depends on:

```julia-auto
(Package1) pkg> up
(Package1) pkg> add X
(Package1) pkg> add Y
(Package1) pkg> add Z

```

These `add` operations affect the `Package1/Project.toml` file. Note that I first did a did a `pkg> up` to update Julia’s package registries - just in case.

If you wish to see what packages are available to this `Package1`, run the package status command:

```julia-auto
(Package1) pkg> st

```

And you might want to return to the default environment aftwerwards:

```julia-auto
(Package1) pkg> activate
pkg>

```

### Is it the “right” package?

> [@felix](#):
>
> **and not just fall back on whatever globally available packages there are**.

Not possible. Try importing a package from `Package1` that isn’t in its `Project.toml` file:

`Package1/src/Package1.jl`:

```julia-auto
module Package1
	import NumericIO
end

```

Now from Julia command line:

```julia-auto
julia> using Package1
┌ Warning: Package Package1 does not have NumericIO in its dependencies:
│ - If you have Package1 checked out for development and have
│ added NumericIO as a dependency but haven't updated your primary
│ environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with Package1
└ Loading NumericIO into Package1 from project dependency, future warnings for Package1 are suppressed.

```

But to ensure you have the CORRECT version of X (correct SHA), you might also want to save `Package1`’s Manifest.toml file (which could include path to the _ **intended version** _ of X).

### Accessing `Package1`’s `X` package from another module

> [@felix](#):
>
> - how do you load a given package in a foreign module?

You don’t. Whatever is loaded in Juila is available _ **everywhere** _. Julia doesn’t do information hiding.

All you really need is a reference to that module. That’s super easy:

`MyForeignModule/src/MyForeignModule.jl`:

```julia-auto
module MyForeignModule
import Package1
const X = Package1.X #That's it! You've got a reference to the CORRECT X package!

X.dosomething(3) #How to call something with Package1.X!

...
end #module MyForeignModule

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [January 5, 2021, 6:23am UTC](https://discourse.julialang.org/t/loading-packages-programmatically-in-a-different-module/52777/5 "2021-01-05T06:23:33Z")

</div>

> [@felix](#):
>
> the problem remains of making the (right versions of) packages available to the module `M` .

Note that you can only load a single version of a package in any given Julia session, so I am not sure that this is the right way to go. IMO it is best to create a consistent `Project.toml` file an leave everything else to the resolver and the loader.