# Suggestion: Pkg inherit-manifest (similar to "Test-specific dependencies")

**URL:** <https://discourse.julialang.org/t/suggestion-pkg-inherit-manifest-similar-to-test-specific-dependencies/121404>\
**Category:** Internals & Design\
**Created:** [October 17, 2024, 1:53am UTC](https://discourse.julialang.org/t/suggestion-pkg-inherit-manifest-similar-to-test-specific-dependencies/121404 "2024-10-17T01:53:12Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Arlo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arlo/32/211461_2.png) [@Arlo](https://discourse.julialang.org/u/Arlo)\
**Post date:** [October 17, 2024, 1:53am UTC](https://discourse.julialang.org/t/suggestion-pkg-inherit-manifest-similar-to-test-specific-dependencies/121404/1 "2024-10-17T01:53:12Z")

</div>

I’m new to Julia, but have been a software engineer for a while working in various languages. I’ve been studying how Julia Pkg behaves in regards to dependency versions, manifests, and test target. I think there’s a need for Pkg `develop` with manifest inheritance.

This is a similar topic to [Is there concept of dev dependencies in Pkg? - #10 by andre1sk](https://discourse.julialang.org/t/is-there-concept-of-dev-dependencies-in-pkg/46908/10)

# Sandbox dev problem

Our team uses this sandbox development workflow to add useful development packages, and generally works in this environment.

```julia-auto
$ mkdir sandbox
$ cd sandbox
$ julia --project=.

julia> ]
(sandbox) pkg> dev ../
(sandbox) pkg> add Revise Infiltrator BenchmarkTools

```

We commit the root project’s Manifest.toml so that all developers are working with the exact same versions. The problem is the `sandbox` has a separate manifest, so pkg versions can differ (within the limitations of compat) depending on when the developer last did an `update` in their sandbox. This negates the benefits of committing the Manifest file as now our developers may be working with different package versions.

# Test-specific dependencies behavior

The version behavior I want already exists in Julia, but only for `] test`

> **[Alternative approach: test/Project.toml file test specific dependencies - 5....](https://pkgdocs.julialang.org/v1/creating-packages/#Alternative-approach:-test/Project.toml-file-test-specific-dependencies)**
>
> Documentation for Pkg.jl.

This is underdocumented (maybe I should add these notes to the docs?), but `test` will always use the package versions of the root Manifest, even when a package is added to the test project.

For example, if you have:

- root Project `[compat] Example = "0.5"`
- `Example v0.5.4` in root manifest
- `Example v0.5.5` in test manifest

When you run `] test`, you will see this warning and `Example` will be v0.5.4 in both runtests.jl and the root project’s module.

> Warning: Entry in manifest at “C:\Users\awhite\code\playground\TestTest\test” for package “Example” differs from that in “C:\Users\awhite\code\playground\TestTest\Manifest.toml”

Note: a similar problem to sandbox here is if you activate the test environment separately, this special test context and manifest version behavior is lost.  
`julia --project=test`

```julia-auto
julia> using Example; pkgversion(Example)
v"0.5.5"

```

# Proposed Solution

So what I would like is a way to create project environments that have the `test` inherit other manifest versions behavior.

`] dev --inherit-manifest ..`

Though one problem is what if you do this with multiple packages? So maybe it should be a feature of Project.toml `inherit_manifest = ".."`

Also, it seems like it would be better if test/Project.toml used this feature, so the environment you have when activating (`julia --project=test`) is the same as `] test`. And since test doesn’t use `dev ..` this is an argument for Project.toml feature.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [October 17, 2024, 2:26am UTC](https://discourse.julialang.org/t/suggestion-pkg-inherit-manifest-similar-to-test-specific-dependencies/121404/2 "2024-10-17T02:26:50Z")

</div>

Would workspaces solve your issue?

# Pkg v1.12 Release Notes

- Pkg now has support for “workspaces” which is a way to resolve multiple project files into a single manifest. The functions `Pkg.status`, `Pkg.why`, `Pkg.instantiate`, `Pkg.precompile` (and their REPL variants) have been updated to take a `workspace` option. Read more about this feature in the manual about the TOML-files.

???

---

<div class="post-metadata">

**Author:** ![Arlo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arlo/32/211461_2.png) [@Arlo](https://discourse.julialang.org/u/Arlo)\
**Post date:** [October 17, 2024, 3:36am UTC](https://discourse.julialang.org/t/suggestion-pkg-inherit-manifest-similar-to-test-specific-dependencies/121404/3 "2024-10-17T03:36:09Z")

</div>

Pkg dev documentation:

> **[10. Project.toml and Manifest.toml · Pkg.jl](https://pkgdocs.julialang.org/dev/toml-files/#The-%5Bworkspace%5D-section)**
>
> Documentation for Pkg.jl.

> When the package manager resolves dependencies, it considers the requirements of all the projects in the workspace. The compatible versions identified during this process are recorded in a single manifest file located next to the base project file.

yes, this sounds like what we want. thanks!

* * *

I’m curious if the recommendation with Pkg 1.12 will be to use workspaces for `test` and deprecate the [target based test specific dependencies](https://pkgdocs.julialang.org/dev/creating-packages/#target-based-test-specific-dependencies). The sample `[workspace]` code seems to imply this.

---

<div class="post-metadata">

**Author:** ![Arlo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arlo/32/211461_2.png) [@Arlo](https://discourse.julialang.org/u/Arlo)\
**Post date:** [October 17, 2024, 3:46am UTC](https://discourse.julialang.org/t/suggestion-pkg-inherit-manifest-similar-to-test-specific-dependencies/121404/4 "2024-10-17T03:46:41Z")

</div>

One concern with workspaces is we git ignore _sandbox_ and developers may add different packages, yet they would all modify the central manifest, which we commit. So this could be a bit annoying. You’d need to be careful not to commit Manifest changes caused by your sandbox.

Or maybe it would be ok, the manifest would just end up accumulating all the various sandbox packages developers have added, but that feels wrong since there’s no way to recreate the Manifest since we don’t commit _sandbox/Project.toml_

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [October 17, 2024, 5:54am UTC](https://discourse.julialang.org/t/suggestion-pkg-inherit-manifest-similar-to-test-specific-dependencies/121404/5 "2024-10-17T05:54:37Z")

</div>

Why do you commit the Manifest.toml in the first place?

I would just commit the Project.toml file, perhaps `freeze()` the versions and ask the developers to update their packages every morning.

The `freeze()` function is provided by:

> **[GitHub - ufechner7/PkgHelpers.jl: Helper functions for the Julia package manager](https://github.com/ufechner7/PkgHelpers.jl)**
>
> Helper functions for the Julia package manager

---

<div class="post-metadata">

**Author:** ![Arlo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arlo/32/211461_2.png) [@Arlo](https://discourse.julialang.org/u/Arlo)\
**Post date:** [October 21, 2024, 12:08am UTC](https://discourse.julialang.org/t/suggestion-pkg-inherit-manifest-similar-to-test-specific-dependencies/121404/6 "2024-10-21T00:08:52Z")

</div>

Committing _Manifest.toml_ ensures we’re all using the exact same versions. Ideally, a minor update wouldn’t introduce bugs, but it happens sometimes. It also ensures package versions aren’t changing when you build and deploy. (in Docker containers)

The other nice thing about it is you don’t need to tell everyone to `update` after _Project.toml_ changes; everyone’s Manifest is managed for them. Though you do need to tell people to git merge/rebase their working branches if you’ve pushed a Manifest change and they want to use the latest packages.

Also, if you’re using a different Julia version, it will warn you, which is a nice reminder to `juliaup update`, or that the Manifest needs to be updated if you’re on latest.

---

<div class="post-metadata">

**Author:** ![caleb-allen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/caleb-allen/32/14054_2.png) [@caleb-allen](https://discourse.julialang.org/u/caleb-allen)\
**Post date:** [October 21, 2024, 8:14am UTC](https://discourse.julialang.org/t/suggestion-pkg-inherit-manifest-similar-to-test-specific-dependencies/121404/7 "2024-10-21T08:14:09Z")

</div>

I’ve done work on projects that use this approach (committing `Manifest.toml`) and I understand that sometimes it’s the right solution.

Still, I wonder if a strict compat policy would achieve the outcome you’re looking for?

> [@Arlo](#):
>
> Or maybe it would be ok, the manifest would just end up accumulating all the various sandbox packages developers have added, but that feels wrong since there’s no way to recreate the Manifest since we don’t commit _sandbox/Project.toml_

I think that by (understandably) orienting the workflow around `Manifest.toml`, you’re encountering these second order problems related to machine-specific installations and their histories; `Project.toml` exists for precisely this reason, many projects can be combined into one Manifest, allowing each Project to encompass _only_ the scope of its own code; a workflow such as your `sandbox/` workflow leaves the outer `Project.toml` unaffected. Using the `freeze()` solution @ufechner7 mentioned would get you pretty close to where you are with a version-controlled `Manifest.toml` while also giving you the flexibility of `Project.toml`. The more your project grows the more difficult upgrades (of packages and/or the language) will be, and the more valuable Pkg features like `compat` will be across packages.

Lastly, you could continue checking in a `Manifest.toml` for production _only_ (and CI if you’ve got it), but leave them out to use `Project.toml`s for development, avoiding the problem of “sandbox accumulation”.

---

<div class="post-metadata">

**Author:** ![Arlo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arlo/32/211461_2.png) [@Arlo](https://discourse.julialang.org/u/Arlo)\
**Post date:** [October 22, 2024, 7:42am UTC](https://discourse.julialang.org/t/suggestion-pkg-inherit-manifest-similar-to-test-specific-dependencies/121404/8 "2024-10-22T07:42:25Z")

</div>

Regarding the root Project.toml, I don’t understand the benefit of strict compat policy (freeze) via compat VS committing Manifest.toml.

If you freeze your Project.toml, doesn’t that increase the risk of dependent packages not being able to resolve a compat solution?  
_This is the core issue, that for most dependent projects, you want normal flexible compat resolution, but for projects like test and sandbox, you want strict compat or a feature that uses the other project’s manifest._

And when you want to `update`, now you need to clear out your compat section, but have no idea what compat entries were there for the purpose of freezing versus defining an actual compat requirement like `"^1.1"`.

I suppose one of the headaches with comitted Manifest is when minor Julia versions are released, but isn’t this why versioned Manifest file support in Julia 1.11?

* * *

> [@caleb-allen](#):
>
> avoiding the problem of “sandbox accumulation”.

This was hypothetical for Julia 1.12 workspaces where our sandbox would be defined as a workspace. We gitignore the sandbox now, but in 1.12 it would affect the root Manifest (version controlled) if we made it a workspace _(if I understand the workspace vision correctly)_. So not sure if we would want to make our sandbox a workspace in 1.12 or not.

* * *

Back to the ideal sandbox question. I can’t think of a good solution that isn’t awkward. The best I can think of is to freeze the root Project.toml temporarily, then run Pkg commands in your sandbox like add, update, etc, then revert changes to root Project.toml. Every developer needs to do this and `resolve` whenever the root Project is updated.

---

<div class="post-metadata">

**Author:** ![Arlo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arlo/32/211461_2.png) [@Arlo](https://discourse.julialang.org/u/Arlo)\
**Post date:** [October 22, 2024, 7:58am UTC](https://discourse.julialang.org/t/suggestion-pkg-inherit-manifest-similar-to-test-specific-dependencies/121404/9 "2024-10-22T07:58:40Z")

</div>

To reiterate the dev workflow benefit of committing Manifest.toml: developers don’t need to do anything to sync their packages other than restart their Julia session. However, with the sandbox pattern, they need to remember to `] update` to stay in sync.

With the suggested inherit-manifest feature, it would be nice if the experience was the same when working in the sandbox environment. When the root Manifest updates (via git), you restart your sandbox Julia session, and the package versions from root Manifest are used without needing to `resolve`.