How to dynamically import packages and run their functions from a setting file

General Usage
1

I am trying to create a set of packages where I have a main package that know nothing of subpackages but it call a certain function of them, and where the subpackages to use are defined in a configuration file.

So, I have a main package:

*MainPackage

  • run()

And several “optional” packages:

OptionalPackage_regionA

  • foo!()

OptionalPackage_regionB

  • foo!()

The user should be able to just use the main package and call the run() function:

using MainPackage
MainPackage.run()

Where the run function should do something like that:

  • load a setting toml file and get regions_to_run (["regionA","regionB"]) and settings (a dict)
  •  if OptionalPackage_R in current environment
     import OptionalPackage_R
     run OptionalPackage_R.foo!(settings)
 else
     raise an error
 end

I am doing this because the different regions implement things in a different way, but I don’t want the main package to have a priori list of possible regions.

Perhaps with extensions, but there the user has first to type before the run using OptionalPackage_regionA; using OptionalPackage_regionB and then the main package, at least on its Project.toml file, needs to know about these regions.

Suggestions ?

2

I don’t exactly understand what you want. Do you want to load packages conditionally, based on a parameter file? You might need @eval to do that (a symptom that there is probably a better way to accomplish your goals). You might find using Pkg; Pkg.project().dependencies to be useful for finding what is in your project (although this may not be the best way – this kind of introspection isn’t super-well supported because it’s not usually very useful). If you’re thinking about this, I’ll discourage it and say that there’s probably a better way. If you elaborate a little bit, someone can probably suggest it.

In any case, your MainPackage does not need to import, depend on, or even know about your optional packages. More idiomatically, your MainPackage might define an interface (a set of abstract types and/or functions) and your optional packages would extend it with their custom functionality. So your optional packages would depend on your main package (not so onerous when they intend to implement/extend its functionality), but not the other way around.

An example of the pattern I would propose is given in this post. This is basically the same way any interface in Julia is extended (for example, how one implements a custom AbstractArray) except that in this case the dispatch is based on a token (that may hold no data) instead of some data-carrying object (like an AbstractArray).

3

The difference with your approach and what I am trying to achieve is that in your approach it is the user of MainPackage/optional packages to make the call, while in my needs it is MainPackage itself that needs to make a call to the optional packages, where the name of the package is in some configuration file (as a string).

I think what I am looking for is a sort of session-specific Project/Manifest files.
Normally Project/Manifest files are persistent, while I need a version that exists only for the time of a session.

4

There isn’t a concept of a session-specific project file (that I’ve ever heard of). It just doesn’t really fit with Julia’s concept of a project. You can use Pkg to assemble one programmatically, I guess, but I don’t think you’re doing yourself any favors. Is it such a problem to have them all in one larger project and only utilize a subset in a given session? Why do they need to be loaded only conditionally?

Your desired ultra-simple using MainPackage; MainPackage.run() just isn’t practical (especially not without @eval but maybe even with it) without MainPackage being aware of what it can call.

Still lacking a clear understanding of exactly what you want, I will suggest two possibilities:

  • Make MainPackage depend on all your subpackages. Use your regions_to_run to assemble a Dict/Vector/etc of functions/tokens that you’ll actually use.
  • MainPackage does not depend on the subpackages, but the subpackages depend on MainPackage. The subpackages register themselves with MainPackage in their __init__() by calling MainPackage.register_functionality("regionA", functionality_for_A) so that when MainPackage wants "regionA", it uses functionality_for_A (a function, dispatch token, etc) to implement it. Your regions_to_run can be used to subselect the registered functionalities into an active list for the actual run.

In the latter version, you would instead call

using MainPackage
using PackageForRegionA, PackageForRegionB
MainPackage.run()

where run() will read the config file, assemble the region processing subset, and throw an error if an unrecognized processing mode is requested (e.g., if you request "regionZ" but you haven’t registered functionality for it – probably by using PackageForRegionZ).

1 Like