More convenient way to develop dependent package?

General Usage
1

I’ve run into this problem alot over the years, and finally figured I’d ask if there’s not a solution I’m missing.

The setup is that I have some base package which I’m developing, call it BasePackage. Then I’m working on a project which uses it, call it MainProject. This is an active “top-level” project with some colleagues so I want everything exactly reproducible so this has a Manifest.toml checked into the repo so everyone can just instantiate and go.

As I work on MainProject, I may discover changes I need to make to BasePackage. When this happens, I have to dev BasePackage into my MainProject working directory, make the changes, commit them, then look up the commit hash for that commit, then add BasePackage#that_hash so the Manifest.toml in MainProject records the new desired commit.

Its this last set of steps that’s always a pain. Obviously as thing stabilize this happens less, but I’m tending to work in a fast developing code-base, so to me this is super frequent. Worse, constantly switching between adding and deving BasePackage screws up Revise so I have to restart my session every time.

Is there maybe some better way to do this?

2 Likes
2

Is the repo for BasePackage public? Perhaps you could add BasePackage#master to your Project.toml file, and have users run up everytime they instantiate the project? This will update the Manifest accordingly and automatically.

(I think this should work, but I am no expert in package management).

3

I regularly have the same situation at work with the difference that BasePackage is registered in a private registry. My workflow is approximately the same:

  1. dev BasePackage in MainProject.
  2. Make changes in BasePackage and bump the version number in Project.toml.
  3. Commit, push, and let CI register the new version of BasePackage. (*)
  4. Update to latest version of BasePackage in MainProject.

(*) Before setting up CI for this I did it manually, typically by

$ cd BasePackage
$ julia
julia> using LocalRegistry  # from global environment
julia> register()

At any rate I recommend setting up your own registry if you’re sharing one or more internal packages with colleagues. LocalRegistry is designed to make this easy.

3 Likes
4

Since you are developing based on Git repos, I suppose simply publishing your desired version of BaseProject under a stable branch, and let your MainProject depend on that branch could do the work.

You can have a local copy of MainProject which uses a deved version of BaseProject and also a copy of MainProject which tracks BaseProject.git#stable. The second copy is only for confirming whether your changes have taken place. In BaseProject, you can edit as you like, which will be immediately seen by any project that has deved it. When you are satisfied, commit your final result to the stable branch, so that your work will be visible to any project that tracks the stable branch after it is updated.

5

Thanks, unfortunately the issue with tracking BasePackage#stable or BasePackage#master or something is that it ceases to be reproducible, since for any past commits in MainProject I won’t know what the right working version of BasePackage was. It also doesn’t really solve the issue that I need to keep switching between adding and deving it, if I understand the suggestion (my situation is that upgrades to BasePackage are happening in unison with MainProject, so its not just an external dependency that occasionally is updating independently).

6

If you do ]add BaseProject#stable, then Manifest.toml will record the commit that it picks. Upload it so that each version of MainProject records what commit it uses. As long as the git repo is accessible, any one can instantiate the environment with the Manifest.toml.

With two copies of MainProject, my suggested workflow would be:

  1. Edit BaseProject, and test in the deved MainProject.
  2. Commit the changes to stable branch.
  3. Pkg.update in the MainProject which tracks BaseProject.git#stable. Now Manifest.toml records the commit in the last step.
  4. Commit Manifest.toml of MainProject. Optionally test before you do so.

For any one who is working on MainProject, run Pkg.instantiate after they have checked a commit to ensure they use the right version of BaseProject.

1 Like
7

If you do this

pkg> add "/path/to/Basepackage"

It gets added to the Project using the UUID

and written to the Manifest using the PATH

and then don’t share the Manifest

8

Ah! This I had not appreciated, thanks, yea that can save me the looking up the hash by hand. But what happens if the user pulls a new version of MainPackage but forgets instantiate? Error message, or silently use the old version, or automatic update?

Your suggested workflow with two copies of MainProject is interesting. I guess the issue will be if I change files in MainProject simultaneously with BasePackage. The former I’d have to commit from the dev’ed copy (since that’s presumably my main working directory), and then in a separate commit I’d Pkg.update from the #stable copy? Still seems not totally ideal.

This for now is not an issue for me, they’re all public packages.

Unfortunately this I think is a requirement since I want every commit fully reproducible.

9

Don’t let your users do any Pkg operations at all and give them scripts which include instantiation. I start mine with

using Pkg
Pkg.activate(@__DIR__)
Pkg.instantiate()
1 Like