Importing packages by directory

[MA_Laforge]

Question

Is there a way to perform a “proper” import using a local path as an argument?:

import CMDimData
import_by_path joinpath("/path/to/CMDimData/subpkgs/EasyPlotMPL")

Otherwise, does someone have a better suggestion given the problem I have below?

Background/Premise

I have a package providing glue code for a bunch of backends:
(not important, but: GitHub - ma-laforge/CMDimData.jl: Parametric analysis/visualization +continuous-f(x) interpolation)

If a calling module wants to load the glue code for PyPlot (Matplotlib), it simply has to “import” the package in subpkgs/EasyPlotMPL.

This way, I can limit the direct dependencies of my module, and avoid having julia add/install packages for all the supported plotting backends.

Hack Solution 1: LOAD_PATH

When I import CMDimData, I could register the subpkgs directory with Julia’s LOAD_PATH variable:

push(LOAD_PATH, joinpath(@__DIR__, "../subpkgs"))

That could work: it would add a new search path for packages to Julia’s session. The calling module could then simply do:

import CMDimData
import EasyPlotMPL #Now visible in the search path

The problem is I am changing Julia’s state at a global level, unbeknownst to the user, so I don’t really like this idea.

Hack Solution 2: include macro

What I am doing now is hacking my way in with include:

macro includepkg(pkgname::Symbol)
	path = realpath(joinpath(rootpath,"subpkgs/$pkgname/src/$pkgname.jl"))
	m = quote
		if !@isdefined $pkgname
			include($path)
			$pkgname.__init__()
		end
	end
	return esc(m) #esc: Evaluate in calling module
end

Note that I have to manually __init__() the module, because only “proper” packages get initialized automatically.

Now, the caller must instead run:

import CMDimData
CMDimData.@includepkg EasyPlotMPL

What I don’t like here is that code gets copied into the caller’s module. That means that each caller module that wants to use my glue code must compile a new instance of it.

On a true package import, packages get pre-compiled somewhere (where??), and each subsequent call to import returns a reference to that same “compiled” instance.

So re-compiling the glue code is probably bad because technically, the types declared in these modules would be distinct from each other. Consider this simple example:

module MA
   import CMDimData
   CMDimData.@includepkg EasyPlotMPL
end
module MB
   import CMDimData
   CMDimData.@includepkg EasyPlotMPL
end

Now technically, MA.EasyPlotMPL.SomeType != MB.EasyPlotMPL.SomeType. This might eventually cause unexpected behaviour.

Any suggestions?

[rdeits]

It seems like https://github.com/JuliaPackaging/Requires.jl would solve the problem. The glue-code example in the readme seems to be exactly what you’re looking for.

[Henrique_Becker]

I have a package that makes available a command-line script for solving an optimization problem and Requires.jl really helped. Using it I can isolate the code that uses CPLEX/Gurobi/GLPK/… (some of which are commercial solvers that the users are not able to get access) and only load it if the user loads the respective package. So no specific solver package needs to go in the dependencies, and the user can use any solver package I support effortlessly just loading it first.

[MA_Laforge]

Requires.jl

Well, I looked into Requires.jl, but I essentially get the same issues (though what triggers the package loading changes slightly).

I tried adding the JSON/Colors example found in Requires.jl.README.md to my CMDimData.jl package:

module CMDimData
   using Requires #Must also be in CMDimData.Project.toml [deps]
   function __init__()
      @require HDF5="f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f" @eval using Cairo
   end
[...]
end

(I’m using packages that are easily found so you to try it for yourself - they don’t make sense in my context, though).

The problem is that, Julia (1.3.1) doesn’t really like you to import “new” packages that aren’t registered in your Project.toml file (even if you load it dynamically with @require):

julia> using CMDimData

julia> using HDF5
┌ Warning: Package CMDimData does not have Cairo in its dependencies:
│ - If you have CMDimData checked out for development and have
│   added Cairo as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with CMDimData
└ Loading Cairo into CMDimData from project dependency, future warnings for CMDimData are suppressed.

And that seems reasonable, I suppose. I could see this might cause issues with Julia having to recompile code if a newly imported module tried to overwrite a function or something.

Circumventing this issue

This is why I decided to find a means to only load my glue code in the calling module. By doing this, you just need to make sure the caller’s environment registers the dependent module (Cairo.jl, in this example) in its own Project.toml file.

That’s easy enough to do. In any case, if she forgets, the user will get a very useful error message informing her of this:

ERROR: ArgumentError: Package Cairo not found in current path:
- Run `import Pkg; Pkg.add("Cairo")` to install the Cairo package.

But the issue still remains

Essentially, I can only include the glue code in the caller module. That way, my glue code can properly import Cairo, because the caller will have added it to her environment’s Project.toml file.

And that means my code has to compiled in each module that tries to use the glue code.

Well, unless I can find a way to properly import my glue code module, instead of simply include ing it.

Unless I am missing something??

[marius311]

@MA_Laforge

The problem is that, Julia (1.3.1) doesn’t really like you to import “new” packages that aren’t registered in your Project.toml file (even if you load it dynamically with @require ):

Just swap using Cairo with using .Cairo (note the dot) to get rid of that warning. Its mentioned at the end of the first section of the Requires readme.

Edit: ah sorry just realized in your example the package you want to using is not the same one being @required. I’m not sure this will work in that case. But you can (and probably should) just @require all the packages you actually need, you can nest them, then it should be fine.

[MA_Laforge]

@marius311

But you can (and probably should) just @require all the packages you actually need, you can nest them, then it should be fine.

I definitely agree wit the part where you said “all the packages”.

That’s basically my problem: I need for Julia to recognize EasyPlotMPL as a package irrespective of whether @require is used or not.

Using my minimal-dependency package

The idea is that I want people to be able to add my core package:

]add https://github.com/ma-laforge/CMDimData.jl

which doesn’t have plotting backends as dependencies, thus avoiding a forced download/install/compile of all of their respective packages. I want users to install the only the backends they choose.

Too many .git repositories

So now, users should import the glue package corresponding to their desired plotting backend - but I don’t want each one of those glue packages to be in their own separate repository:

]add https://github.com/ma-laforge/EasyPlotMPL.jl

Repository overload: too many mini-glue packages!

v1.5 support for subdir

If I stick with Julia 1.5+, I could start using the multi-package repo capbability:

]add https://github.com/ma-laforge/CMDimData.jl:subpkgs/EasyPlotMPL

(See https://github.com/JuliaLang/Pkg.jl/pull/1422, and other PRs with “subdir”).

However, even that is a bit heavy for my purposes. I don’t really want the user to have to ]add my glue packages if they don’t have to.

Adding to LOAD_PATH instead

As I mentioned earlier, CMDimData could just modify Julia’s global LOAD_PATH when it runs. That basically registers my package’s subpkgs directory as a sort of additional package “library”:

push!(LOAD_PATH, joinpath(@__DIR__, "../subpkgs"))

But I don’t like this, as it modifies Julia’s global state in a unsettling way.

Import by path

So the simplest solution to me would be to import a package by specifying the package path.

I suppose I could do the following in an @require statement:

push!(LOAD_PATH, joinpath(@__DIR__, "../subpkgs"))
import EasyPlotMPL
pop!(LOAD_PATH)

But despite that global state is restored, it still seems a bit hacky for a well designed language like Julia.

[Henrique_Becker]

I am not sure if I understand your problem. But I have never used Requires.jl to import some module only if some other module is available. What I do is to check if some module is available, and then, if it is, define methods that make use of the module inside them. If you need a module you have two options: (1) have it in your dependencies; (2) instead of loading some module only if another module is loaded, do something (define functions, types, etc…) only if both are loaded. I think these are the most clean approaches to this problem.

[MA_Laforge]

@Henrique_Becker

if it is, define methods that make use of the module inside them

What you are effectively suggesting here is that I “include” my glue code somehow when I detect my dependent module is present.

That is essentially the solution I am using right now:

@MA_Laforge

Hack Solution 2: include macro

But like I said, that means my glue code needs to be compiled multiple times for each caller module that uses it, so I find it somewhat suboptimal.