Hello, for an experimental idea I would like to try to create and manipulate modules on the fly, including letting them use packages. I’m struggling to understand/get hints from the docs about this, so I wanted to ask for any pointers on the following:
how do you use the Pkg machinery to calculate a set of packages given the deps and compat of a Project.toml file?
how do you load a given package in a foreign module?
Here’s my aspirational pseudo-code marked with # ???, grateful for any corrections or suggestions:
The package interface is meant to be declarative, so this is not part of the exposed API. That said, you can call the resolver, and then load with eval and using.
I am not sure what the purpose is, but this may not be a good idea though.
Thanks, that may be helpful for the resolution part, but the problem remains of making the (right versions of) packages available to the module M.
I looked through Resolve, but couldn’t follow the flow of it from reading. (E.g. none of the functions in Resolve.jl take a dictionary of dependencies or compatibilities, they all want a graph or a solution vector from somewhere else.)
I guess from API.jl that the key resolution action happens in the following (link), which acts on a ctx::Context object:
The key thing is that, when I load the dependencies X,Y,Z inside the module M by using calls to using X,Y,Z in the file includethis.jl which is executed thanks to the call Core.include(M,"includethis.jl"), I want M to access the packages and versions specified through deps and compat specifically for M, and not just fall back on whatever globally available packages there are. Is that controlled by the currently relevant environment/EnvCache? How do I change the “environment” that M operates in?
Do I have to use something like Pkg.activate? All the machinery I have seen seems quite intimately linked to having a particular folder, containing a particular set of files (Project.toml and Manifest.toml), but I can’t tell if all of those big guns are necessary for what I want? I don’t have a folder with those files, so I would have to create a temporary set-up just for that, and even then I still don’t know how to associate it to M.
Hard for me to tell if you are trying to create something special/new for Julia or if you simply don’t now that these things already exist.
Assuming I understand your issue correctly
How to correctly create “software modules”
When you create a new package (basically a “software module”), you should have the following structure:
Package1 #Base directory of your "software module"
├── Project.toml #(required, esp. for what YOU want)
├── Manifest.toml #(optional)
├── src
│ ├── Package1.jl
│ ├── P1SubFile1.jl #If package too big for 1 file (typical).
│ └── P1SubFile2.jl
... #Typically want a "test/" directory, but we'll ignore that
Note that I chose the name “Package1” instead of “Module1” to highlight the fact that “software modules” are packages (and Julia "module"s are namespaces).
Also note that you can generate the skeleton structure using:
julia> ]
pkg> generate Package1
IMPORTANT:
Don’t forget that the code from Package1.jl file must be encased in a “module Package1 … end” block to function properly.
Making a package available
Once this package is generated, you must make it available to your current project (or current package) environment with the help of pkg> add or pkg> dev:
julia> ]
pkg> dev /abs/path/to/Package1
I chose dev over add because, at this stage, you probably don’t want to git commit .; pkg> up Package1 every time you make a change in Package1.
Controlling dependencies
That’s what the Package1/Project.toml file is for. To add dependencies to your package you must first activate this package (which then allows you to edit its Project.toml file):
then you can to add whatever packages Package1 depends on:
(Package1) pkg> up
(Package1) pkg> add X
(Package1) pkg> add Y
(Package1) pkg> add Z
These add operations affect the Package1/Project.toml file. Note that I first did a did a pkg> up to update Julia’s package registries - just in case.
If you wish to see what packages are available to this Package1, run the package status command:
(Package1) pkg> st
And you might want to return to the default environment aftwerwards:
(Package1) pkg> activate
pkg>
Is it the “right” package?
Not possible. Try importing a package from Package1 that isn’t in its Project.toml file:
Package1/src/Package1.jl:
module Package1
import NumericIO
end
Now from Julia command line:
julia> using Package1
┌ Warning: Package Package1 does not have NumericIO in its dependencies:
│ - If you have Package1 checked out for development and have
│ added NumericIO as a dependency but haven't updated your primary
│ environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with Package1
└ Loading NumericIO into Package1 from project dependency, future warnings for Package1 are suppressed.
But to ensure you have the CORRECT version of X (correct SHA), you might also want to save Package1’s Manifest.toml file (which could include path to the intended version of X).
Accessing Package1’s X package from another module
You don’t. Whatever is loaded in Juila is available everywhere. Julia doesn’t do information hiding.
All you really need is a reference to that module. That’s super easy:
MyForeignModule/src/MyForeignModule.jl:
module MyForeignModule
import Package1
const X = Package1.X #That's it! You've got a reference to the CORRECT X package!
X.dosomething(3) #How to call something with Package1.X!
...
end #module MyForeignModule
Note that you can only load a single version of a package in any given Julia session, so I am not sure that this is the right way to go. IMO it is best to create a consistent Project.toml file an leave everything else to the resolver and the loader.