Suggestion: Pkg inherit-manifest (similar to "Test-specific dependencies")

Internals & Design
1

I’m new to Julia, but have been a software engineer for a while working in various languages. I’ve been studying how Julia Pkg behaves in regards to dependency versions, manifests, and test target. I think there’s a need for Pkg develop with manifest inheritance.

This is a similar topic to Is there concept of dev dependencies in Pkg? - #10 by andre1sk

Sandbox dev problem

Our team uses this sandbox development workflow to add useful development packages, and generally works in this environment.

$ mkdir sandbox
$ cd sandbox
$ julia --project=.

julia> ]
(sandbox) pkg> dev ../
(sandbox) pkg> add Revise Infiltrator BenchmarkTools

We commit the root project’s Manifest.toml so that all developers are working with the exact same versions. The problem is the sandbox has a separate manifest, so pkg versions can differ (within the limitations of compat) depending on when the developer last did an update in their sandbox. This negates the benefits of committing the Manifest file as now our developers may be working with different package versions.

Test-specific dependencies behavior

The version behavior I want already exists in Julia, but only for ] test

This is underdocumented (maybe I should add these notes to the docs?), but test will always use the package versions of the root Manifest, even when a package is added to the test project.

For example, if you have:

  • root Project [compat] Example = "0.5"
  • Example v0.5.4 in root manifest
  • Example v0.5.5 in test manifest

When you run ] test, you will see this warning and Example will be v0.5.4 in both runtests.jl and the root project’s module.

Warning: Entry in manifest at “C:\Users\awhite\code\playground\TestTest\test” for package “Example” differs from that in “C:\Users\awhite\code\playground\TestTest\Manifest.toml”

Note: a similar problem to sandbox here is if you activate the test environment separately, this special test context and manifest version behavior is lost.
julia --project=test

julia> using Example; pkgversion(Example)
v"0.5.5"

Proposed Solution

So what I would like is a way to create project environments that have the test inherit other manifest versions behavior.

] dev --inherit-manifest ..

Though one problem is what if you do this with multiple packages? So maybe it should be a feature of Project.toml inherit_manifest = ".."

Also, it seems like it would be better if test/Project.toml used this feature, so the environment you have when activating (julia --project=test) is the same as ] test. And since test doesn’t use dev .. this is an argument for Project.toml feature.

2

Would workspaces solve your issue?

Pkg v1.12 Release Notes

  • Pkg now has support for “workspaces” which is a way to resolve multiple project files into a single manifest. The functions Pkg.status, Pkg.why, Pkg.instantiate, Pkg.precompile (and their REPL variants) have been updated to take a workspace option. Read more about this feature in the manual about the TOML-files.

???

3

Pkg dev documentation:

When the package manager resolves dependencies, it considers the requirements of all the projects in the workspace. The compatible versions identified during this process are recorded in a single manifest file located next to the base project file.

yes, this sounds like what we want. thanks!

I’m curious if the recommendation with Pkg 1.12 will be to use workspaces for test and deprecate the target based test specific dependencies. The sample [workspace] code seems to imply this.

1 Like
4

One concern with workspaces is we git ignore sandbox and developers may add different packages, yet they would all modify the central manifest, which we commit. So this could be a bit annoying. You’d need to be careful not to commit Manifest changes caused by your sandbox.

Or maybe it would be ok, the manifest would just end up accumulating all the various sandbox packages developers have added, but that feels wrong since there’s no way to recreate the Manifest since we don’t commit sandbox/Project.toml

1 Like
5

Why do you commit the Manifest.toml in the first place?

I would just commit the Project.toml file, perhaps freeze() the versions and ask the developers to update their packages every morning.

The freeze() function is provided by:

6

Committing Manifest.toml ensures we’re all using the exact same versions. Ideally, a minor update wouldn’t introduce bugs, but it happens sometimes. It also ensures package versions aren’t changing when you build and deploy. (in Docker containers)

The other nice thing about it is you don’t need to tell everyone to update after Project.toml changes; everyone’s Manifest is managed for them. Though you do need to tell people to git merge/rebase their working branches if you’ve pushed a Manifest change and they want to use the latest packages.

Also, if you’re using a different Julia version, it will warn you, which is a nice reminder to juliaup update, or that the Manifest needs to be updated if you’re on latest.

7

I’ve done work on projects that use this approach (committing Manifest.toml) and I understand that sometimes it’s the right solution.

Still, I wonder if a strict compat policy would achieve the outcome you’re looking for?

I think that by (understandably) orienting the workflow around Manifest.toml, you’re encountering these second order problems related to machine-specific installations and their histories; Project.toml exists for precisely this reason, many projects can be combined into one Manifest, allowing each Project to encompass only the scope of its own code; a workflow such as your sandbox/ workflow leaves the outer Project.toml unaffected. Using the freeze() solution @ufechner7 mentioned would get you pretty close to where you are with a version-controlled Manifest.toml while also giving you the flexibility of Project.toml. The more your project grows the more difficult upgrades (of packages and/or the language) will be, and the more valuable Pkg features like compat will be across packages.

Lastly, you could continue checking in a Manifest.toml for production only (and CI if you’ve got it), but leave them out to use Project.tomls for development, avoiding the problem of “sandbox accumulation”.

8

Regarding the root Project.toml, I don’t understand the benefit of strict compat policy (freeze) via compat VS committing Manifest.toml.

If you freeze your Project.toml, doesn’t that increase the risk of dependent packages not being able to resolve a compat solution?
This is the core issue, that for most dependent projects, you want normal flexible compat resolution, but for projects like test and sandbox, you want strict compat or a feature that uses the other project’s manifest.

And when you want to update, now you need to clear out your compat section, but have no idea what compat entries were there for the purpose of freezing versus defining an actual compat requirement like "^1.1".

I suppose one of the headaches with comitted Manifest is when minor Julia versions are released, but isn’t this why versioned Manifest file support in Julia 1.11?

This was hypothetical for Julia 1.12 workspaces where our sandbox would be defined as a workspace. We gitignore the sandbox now, but in 1.12 it would affect the root Manifest (version controlled) if we made it a workspace (if I understand the workspace vision correctly). So not sure if we would want to make our sandbox a workspace in 1.12 or not.

Back to the ideal sandbox question. I can’t think of a good solution that isn’t awkward. The best I can think of is to freeze the root Project.toml temporarily, then run Pkg commands in your sandbox like add, update, etc, then revert changes to root Project.toml. Every developer needs to do this and resolve whenever the root Project is updated.

1 Like
9

To reiterate the dev workflow benefit of committing Manifest.toml: developers don’t need to do anything to sync their packages other than restart their Julia session. However, with the sandbox pattern, they need to remember to ] update to stay in sync.

With the suggested inherit-manifest feature, it would be nice if the experience was the same when working in the sandbox environment. When the root Manifest updates (via git), you restart your sandbox Julia session, and the package versions from root Manifest are used without needing to resolve.

1 Like