Is it possible to load a subset of a package?

I have run into issues with long loading times for using some of my custom packages, and it appears to be because those packages load a lot of dependencies (as suggested here). But often my packages only need some small subset of the functionality provided by another package (e.g., I only need gradient from ForwardDiff).

So, I have two questions:

  1. Is there a way to only load a subset of a package? (I.e., something like using ForwardDiff: gradient, but only load code related to gradient instead of loading everything in ForwardDiff.)
  2. If so, how can I structure my package code to enable users to only load a subset of the functionality? Submodules?

Thanks in advance.

I know it is unable to load subsets, but someone else can say more on that. I think that’s why big projects like JuliaStats have separate packages for subsets of functionality?

I’ve used Requires.jl to dynamically load code based on other packages being loaded.

1 Like
  1. AFAIK there is no such mechanism.

  2. If a subset of a package makes sense as a self-contained unit of functionality, perhaps consider making it a package itself.

In the future you may be able to have multiple packages in a single repository:

https://github.com/JuliaLang/Pkg.jl/issues/1251

but for now that is WIP.

2 Likes

I think we can write a macro that includes that subset of the code. Some kind of intelligent macro that parses the source code of the package and finds the dependencies of that subset and includes and evaluates it.

It is hard though :grin:

Another approach is to make dynamic dlls using PackageCompiler.

1 Like

Yes, exactly this. I think developing multiple packages in one repository is the best solution. Packages are the mechanism for identifying/naming a chunk of code to either load or not. In that sense loading part of a package is kind of a contradiction in terms — the part you’re loading is itself a package by definition.

8 Likes

Related issue:

https://github.com/JuliaLang/julia/issues/33670