How to distribute a set of private packages

Tooling Package Management
1

At my company, we have a private package registry and a suite of packages we’ve developed. We can’t make these packages public, but we are thinking about sharing the packages with another company. Assuming I can’t give them access to our private registry, what’s the best way for me to bundle up the packages to share with the other company? Should I just export the correct version of each repo, tar it up, and then give them instructions on how to set LOAD_PATH? Or is there a better solution, perhaps using PackageCompiler? Thanks!

3

I give packages to small groups this way. For classes I just put a project on our university git server and tell the students to clone the repo and park it in their LOAD_PATH. You don’t need to make it a formal package for this, but if your collaborators can’t get to your registry, they can’t use pkg to do the install anyhow.

Please follow up and tell us what you did. I am not convinced that what I do is optimal.

1 Like
4

I don’t know how practical this is (depends on how often this is done, to what extent you use the packages internally or just distribute), but: you could always make a registry just for the packages you plan to distribute. That could be a new registry made just for distribution. Or, if you know in advance that you plan to distribute, these packages could have their own registry from the start.

Then distributing the packages would be as easy as emailing someone a copy of that registry.

1 Like
5

I put this issue off for a while, but I’ve recently started working on it again.

The first thing I did was write a script that exported the correct versions of all of our private packages into a packages directory. You can of course put that directory on the LOAD_PATH. The problem was how to get this to interact with the package manager. Our private packages depend on public packages. How do we get the package manager to install those? I suppose I could output a script that calls Pkg.dev on the packages instead of using LOAD_PATH, but our private packages also depend on each other (there’s one top-level package we are trying to distribute, along with its supporting packages). So, we’d have to call Pkg.dev in the correct order, basically working up the dependency tree. I suppose it would work, but it didn’t seem like the right solution.

Rather than distribute a directory of packages, it seems it would be better to distribute a Pkg depot. The user just needs to put the distributed depot somewhere after the first entry in DEPOT_PATH, and then the package manager should just work. Since the depot already contains the registry metadata and the source code for the appropriate versions of the private packages, the user doesn’t need access to our private registry.

I’ve thought of a few issues with it, but I think they can all be solved.

  1. The depot will contain both public and private packages. I think it should be pretty straightforward to strip out any packages that don’t have URLs pointing to our private GitLab instance. Or maybe there’s a way to use the Pkg API to make a depot with only the packages of interest.
  2. The depot will contain metadata for all of the packages in our private registry, not just the ones we’ve decided to distribute. To address this, I think I’ll try following @hendri54’s suggestion. I could have a script that creates a new local registry on the fly, populates it with only the latest versions of the necessary packages, and then incorporates that into a depot that can be distributed.

What do others think? Is distributing a depot a good idea, or is that abusing the depot system? Has anyone thought of other approaches to distributing private packages?

Thanks!

6

One trick I’ve used when building Docker images from private packages, which has similar constraints, is to take advantage of the fact the Manifest.toml can contain relative paths.

Here is an example Dockerfile:

FROM julia:1.5

RUN mkdir /depot
ENV JULIA_DEPOT_PATH="/depot"

ADD API /app/API
ADD PrivatePkg1 /app/PrivatePkg1
ADD PrivatePkg2 /app/PrivatePkg2
ADD PrivatePkg3 /app/PrivatePkg3

RUN ["julia", "-e", "using Pkg; Pkg.activate(\"/app/API\"); Pkg.instantiate(); Pkg.precompile()"]

The idea is to have a package API and your private packages at the same level in the directory and then you add the private packages to the API manifest via

(API) pkg> dev ../PrivatePkg1
(API) pkg> dev ../PrivatePkg2
(API) pkg> dev ../PrivatePkg3

This gets the private packages into your Manifest.toml with relative paths.

Then you set a location for your depot and instantiate your Manifest.toml. This will find the private packages via relative path and put all the General packages into your depot folder. This way, you don’t need to distribute the entire depot, but just the directories holding the private packages.

This was a little less painful for me than trying to give a remote Docker registry (in my case, Azure Containers) access to a private registry.

I hope this helps.

4 Likes
7

Thanks for the explanation! I think I gave up on the dev approach too quickly. I’ll give it another shot.

1 Like
8

I was able to get the dev approach to work! Thanks again for the help on that.

I’m still curious about the depot approach, though, if anyone has thoughts on that.

1 Like
9

I think that the best approach is a private registry, as suggested by @hendri54. Otherwise, distributing improvements and bugfixes will be tedious.

1 Like
10

You mean host a private registry and then grant the other company access? Or are you saying there’s a way to bundle up a registry and the associated package code?

I agree that hosting a private registry and then just limiting access would be ideal, but I think my company is going to force me to distribute the packages as a zipped up bundle.

11

Maybe https://github.com/GunnarFarneback/LocalRegistry.jl can help? (Haven’t used it myself but looks relevant)

12

Yes.

You can do that, and then extract and Pkg.dev the path to each package. Not ideal, but works.

13

You can register your packages with file:// URLs. It should be possible to make this work if the bundle is unpacked at a standardized path. Otherwise you could deliver a script that creates a registry on site after unpacking your bundle.

2 Likes
14

If the packages live in private github repos, would it be possible to give the other company access to those repos (perhaps through a security token; I don’t know enough about github to figure out exactly how this might work)?
This is essentially the way I run my code on our HPC. Once the registry is installed on the HPC, I don’t need to distribute any code.

15

That’s a cool idea, I’ll give that a try, thanks!