# About checking a package/library \`Manifest.toml\` into control version (again)

**URL:** <https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490>\
**Category:** General Usage\
**Tags:** package, dependencies, versions, environment, manifest\
**Created:** [March 29, 2025, 4:56pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490 "2025-03-29T16:56:01Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![iago-lito](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/iago-lito/32/31924_2.png) [@iago-lito](https://discourse.julialang.org/u/iago-lito)\
**Post date:** [March 29, 2025, 4:56pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/1 "2025-03-29T16:56:01Z")

</div>

Hello @Julia,

From what I understand [here](https://stackoverflow.com/questions/58996197/when-should-a-julia-project-have-a-manifest-and-project-file) and [there](https://discourse.julialang.org/t/does-manifest-toml-belong-in-the-repository/12029) (but really [anywhere](https://github.com/SciML/DifferentialEquations.jl/issues/1080#issuecomment-2758593194)), there is almost dogmatic community consensus that `Manifest.toml` files _may_ be checked into version control for end [applications](https://pkgdocs.julialang.org/v1/glossary/), but _certainly not_ for [packages](https://pkgdocs.julialang.org/v1/glossary/) supposed to be consumed library-style by other julia projects downstream.

However, the [more](https://docs.julialang.org/en/v1/manual/code-loading/#Environments) I [read](https://pkgdocs.julialang.org/v1/compatibility/) about the meaning of the manifest, the less I agree or even _understand_ this consensus. I would like to discuss it again.

Although I understand the choice not to track the `Manifest.toml` (arguably a non-source file), I do not understand that we actually _recommend_ against tracking it. IIUC here are the reasons most frequently invoked in this respect:

* * *

Reason 1: _Library consumers need maximum flexibility on your dependencies versions so their resolver can construct an environment satisfying all their constraints._

This is totally true, but irrelevant to the topic IMO.  
The manifest file is used during an `] instantiate` procedure to reproduce the exact same dependencies environment that the one it was created within. However, library consumers are not expected to `] instantiate` my project. Instead, they are expected to `] add MyPackage`, an action which reads from my `Project.toml` files, its `[deps]` section in particular, and then calls into the resolver to satisfy the constraints listed in the `[compat]` section. IIUC the manifest file is _not even actually used_ during this process.  
Therefore, although I think Reason 1 is a good argument against being overly strict in the `[compat]` section of my project file (_e.g._ against using strict requirements like `"Dep" = "=1.2.3"`), I fail to see how it is an argument against checking in my manifest file, which is (again, IIUC) actually _ignored_ by all downstream consumers of my package.

BTW: I may be wrong in this reasoning, but then I’d need to report that how the manifest is used (or not) is not exactly explicit from the official documentation pages: [[1]](https://docs.julialang.org/en/v1/manual/code-loading/#Environments) [[2]](https://pkgdocs.julialang.org/v1/compatibility/). The above are just the conclusions I came to myself, and confirmed at least [there](https://julialang.zulipchat.com/#narrow/channel/137791-general/topic/Does.20the.20Manifest.20actually.20restricts.20versions.20for.20downstr.2E.2E.2E/near/508913076).

* * *

Reason 2: _Checking in the manifest file yields CI breakage or misleading CI success._

Indeed, FWIU, the following command, typically used in CI:

```julia
] test

```

is actually a synonym of:

```julia
] instantiate; test # Build local environment before testing.

```

Therefore, if my project repo contains a `Manifest.toml` file, then _that_ file will be used to specify the environment. And the tests will be run in an locked/frozen environment that differs from the one actually used by downstream users supposed to `] add MyLib` later, because they would typically get a fresher environment with numerous dependencies bumped.

However, it seems super-easy to avoid that:

```julia
] update; test # Bump local environment before testing.

```

If I check in my manifest file, then I can easily do both:

```julia
] instantiate; test # Check that nothing has broken in the code.
] update; test # Check that nothing has broken in the ecosystem.

```

Or (if it’s too long):

```julia
] update; test # Check both at the same time.

```

Being able to choose is a clear advantage to having the manifest checked in IMO. Especially because the manifest makes it possible to **debug past build** and **bisect versions history** without the ecosystem changes getting in the way.

* * *

Reason 3: _It creates merge conflicts._

It certainly does.  
However, for the same reason you are not supposed to write manifest files as a human, you are not supposed to resolve manifest conflicts as a human.  
If you stumble accross an unmerged manifest file, then you must be currently working on development versions of your code. Therefore, I suggest that it is not a problem that you just delete it, make sure you resolve (actual) conflicts within the `Project.toml`, and then `] update` to automatically generate a fresh, “merged” one.

* * *

Reason 4: _It increases maintenance burden._

I personally see no harm in running `] update; test` every so often and committing the resulting changes to my `Manifest.toml` should the tests still pass. I guess this is also an easy CI automation.

* * *

All in all, and unless I am missing something obvious, I would personally decide to **keep checking manifests** of my julia packages/libraries into control version.

If you also agree with my reasoning, I would maybe further suggest that we start working, as a community, against this unsettling _de-facto_ dogma. First by clarifying the documentation pages in this respect. Then possibly by issuing official recommendations and tradeoff on the same pages.

Julia being used a lot as a scientific tool, anything improving **reproducibility** is desired, and I think manifest files can help a lot in this respect. We should use them more. Libraries/Packages should not be excluded from this effort.

What do you think? : )

* * *

[some more context]:  
Almost two years ago, there has been [a change](https://blog.rust-lang.org/2023/08/29/committing-lockfiles.html) in the official recommendations in rust community regarding this very topic. The result is that users, instead of being recommended to not check in their manifest for libraries, are now recommended to do [whatever suits their project best](https://doc.rust-lang.org/nightly/cargo/faq.html#why-have-cargolock-in-version-control). Julia is not rust though. In particular, Julia cannot have two different versions of the same package within the same environment, whereas two different versions of the same crate [can coexist](https://doc.rust-lang.org/cargo/reference/resolver.html) within the same rust build. IIUC this makes dependency resolution harder for Julia, which needs to pick only _one_ version for every dependency and yet still satisfy every requirement. However, I think that all the above argument still holds.

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [March 29, 2025, 5:09pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/2 "2025-03-29T17:09:38Z")

</div>

I’m not checking in `Manifest.toml` files for libraries because libraries should work with a wide range of versions of their dependencies. So the versions that would be pinned by a manifest do not matter, and accordingly, I wouldn’t spent any effort or mental overhead on this. In fact, it’s probably beneficial to “randomly” have different versions of your dependencies during day-to-day development, and specifically to have “latest” versions of the dependencies appear automatically. As a developer, that mirrors the experience the _user_ of the packages has, which seems like a very good idea.

This is not to say that you don’t also sometimes want to test against specific versions of dependencies. For example, I usually [explicitly test compatibility with the lowest-version-bound of all dependencies](https://github.com/JuliaDocs/DocumenterCitations.jl/blob/74e2c2cc87a03a30bfd30eb822b10f24b8d1d2cb/.github/workflows/ci.yml#L56). But that’s something that’s best done by [pinning those versions in CI](https://github.com/JuliaDocs/DocumenterCitations.jl/blob/74e2c2cc87a03a30bfd30eb822b10f24b8d1d2cb/.github/workflows/ci.yml#L71), not by committing a Manifest.

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [March 29, 2025, 7:33pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/3 "2025-03-29T19:33:23Z")

</div>

In more detail,

> [@iago-lito](#):
>
> Reason 1: _Library consumers need maximum flexibility on your dependencies versions so their resolver can construct an environment satisfying all their constraints._
> 
> This is totally true, but irrelevant to the topic IMO.

You are correct that checking in your `Manifest.toml` has no influence on the users of your package. But that’s not a reason to check in the Manifest.

> the manifest is used (or not) is not exactly explicit from the official documentation pages […] [Your tests will respect the manifest when resolving versions](https://julialang.zulipchat.com/#narrow/channel/137791-general/topic/Does.20the.20Manifest.20actually.20restricts.20versions.20for.20downstr.2E.2E.2E/near/508913076)

Although I would trust @ericphanson’s knowledge on this, that’s completely undocumented (first time I’ve even heard about that), and I’m not sure whether it can be relied upon. Also, I think test dependencies should be specified in [`test/Projects.toml`](https://pkgdocs.julialang.org/v1/creating-packages/#Alternative-approach:-test/Project.toml-file-test-specific-dependencies), not the [`[extras]` section](https://pkgdocs.julialang.org/v1/creating-packages/#target-based-test-specific-dependencies). However, I have no idea how a main `Manifest.toml` and a `test/Manifest.toml` would interact (and the [docs explicitly say](https://pkgdocs.julialang.org/v1/creating-packages/#target-based-test-specific-dependencies) that this is undefined at the moment). I would actually _love_ if `Pkg.test` would take into account the `test/Manifest.toml` file, but that has little bearing on whether you should check in that file.

> [@iago-lito](#):
>
> Reason 2: _Checking in the manifest file yields CI breakage or misleading CI success._

It absolutely produces misleading CI success, and that’s probably the major reason you shouldn’t do this. You want to test the experience a _user_ has when installing the package. That fact that it’s _relatively_ easy to update the Manifest doesn’t mitigate that in any way.

> [@iago-lito](#):
>
> Reason 3: _It creates merge conflicts._

Yeah, this is probably not a major reason not to commit the Manifest

> [@iago-lito](#):
>
> Reason 4: _It increases maintenance burden._

That’s another major reason not to commit the Manifest. I’ve gone down the rabbit hole of custom dev-workflows myself, and it doesn’t scale once you start maintaining more than a very small handful of packages. A lot of these custom workflows were due to my lack of experience, but also a general lack of documentation and tooling a few years ago. However, things have gotten _a lot_ better reently. The [`[sources]` feature](https://pkgdocs.julialang.org/v1/toml-files/#The-%5Bsources%5D-section) in 1.11 is a _major_ step forward, and the [`[workspace]` feature](https://pkgdocs.julialang.org/dev/toml-files/#The-%5Bworkspace%5D-section) in the upcoming 1.12 seems like be the last piece of the puzzle. With that, [my plan](https://github.com/goerz/cookiecutter-juliapackage) would be to use as much “standard” tooling with as little customization as possible.

> [@iago-lito](#):
>
> I would maybe further suggest that we start working, as a community, against this unsettling _de-facto_ dogma. First by clarifying the documentation pages in this respect.

I agree that `Pkg.test` is severely under-documented, and that we should both figure out and document how `Manifest.toml` and `test/Manifest.toml` interact with `Pkg.test`. IMO, both of them _should_ interact: Ideally, `Pkg.test` would use `Manifest.toml` or `test/Manifest.toml` according to the same logic it uses to decide where to get the test dependencies.

However, this does not mean we should encourage people to start committing `Manifest.toml` files.

> Julia being used a lot as a scientific tool, anything improving **reproducibility** is desired, and I think manifest files can help a lot in this respect.

Manifests for _packages_ have nothing to do with reproducibility. Where you want Manifests is _projects_, like sets of Jupyter/Pluto notebooks or [DrWatson](https://juliadynamics.github.io/DrWatson.jl/dev/)-based repos. That is, the actual research documents _using_ Julia packages.

The [Rust discussion](https://blog.rust-lang.org/2023/08/29/committing-lockfiles.html) does raise an interesting point about the reproducibility of CI runs and the potential help with `git bisect`. It might be nice if the [julia-runtest](https://github.com/julia-actions/julia-runtest) could capture the `Manifest.toml` file that `Pkg` used and attach it to the CI run as a artifact. Beyond that, I don’t think the benefit of bisectability with pinned versions outweighs the limitation of not testing/developing packages on the same footing as users the package experience.

---

<div class="post-metadata">

**Author:** ![dilumaluthge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dilumaluthge/32/29283_2.png) [@dilumaluthge](https://discourse.julialang.org/u/dilumaluthge)\
**Post date:** [March 29, 2025, 10:13pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/4 "2025-03-29T22:13:21Z")

</div>

> Your tests will respect the manifest when resolving versions.

This is a bit of a subtle point, but technically speaking this is not 100% guaranteed by default.

If you want to guarantee that Pkg will respect the manifest when running your package’s test suite, you need to do `Pkg.test(; allow_reresolve=false)`.

---

<div class="post-metadata">

**Author:** ![iago-lito](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/iago-lito/32/31924_2.png) [@iago-lito](https://discourse.julialang.org/u/iago-lito)\
**Post date:** [March 30, 2025, 10:29am UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/5 "2025-03-30T10:29:05Z")

</div>

Hello @goerz @dilumaluthge and thank you for discussion : )

> [@goerz](#):
>
> checking in your `Manifest.toml` has no influence on the users of your package. But that’s not a reason to check in the Manifest.

Indeed. I wouldn’t check it in just because I could ^ ^ I do it because I see _benefit_ in it (see below). At least we agree that “Reason 1” is a misleader. It denotes a misunderstanding what manifests are, which I think possibly reveals under-documentation.

> [@goerz](#):
>
> > [Your tests will respect the manifest when resolving versions](https://julialang.zulipchat.com/#narrow/channel/137791-general/topic/Does.20the.20Manifest.20actually.20restricts.20versions.20for.20downstr.2E.2E.2E/near/508913076)
> 
> [..] that’s completely undocumented (first time I’ve even heard about that), and I’m not sure whether it can be relied upon.

> [@dilumaluthge](#):
>
> This is a bit of a subtle point, but technically speaking this is not 100% guaranteed by default.

This is.. rather unsettling. Do I understand correctly that the two following commands may not yield the same result?

```julia
] instantiate; test # Test within environment pinned by Manifest.toml if this exists.
] test # Test within.. *some* environment, not exactly specified which. Undocumented why.

```

My gut feeling is that this is worth an issue against julia or julia docs, right? I’d be happy to (at least) report.

> [@goerz](#):
>
> Also, I think test dependencies should be specified in [`test/Projects.toml`](https://pkgdocs.julialang.org/v1/creating-packages/#Alternative-approach:-test/Project.toml-file-test-specific-dependencies), not the [`[extras]` section](https://pkgdocs.julialang.org/v1/creating-packages/#target-based-test-specific-dependencies).

Yeah, I have also read this part. TBH I have found this so confusing that I am now actually wanting to _revert_ from our `test/Project.toml` to some `[extras]` section instead, just to be sure what julia actually does when I `] test`. IIUC you are agreeing that this is slightly _off-topic_ for this thread. But I’m happy to discuss this in another one, or to find good reading on the topic if you happen to have some to share.

> [@goerz](#):
>
> [checking in the manifest file] absolutely produces misleading CI success [..] That fact that it’s _relatively_ easy to update the Manifest doesn’t mitigate that in any way.

I disagree on that. I think it does mitigate. My argument is that if you care about the meaning of CI success, then you are likely a lib maintainer. Therefore, you’ll be easily explained that the following three simple commands have different semantics, and how/whether to use them:

```julia
] test # Test the code in some.. undocumented, possibly non-reproducible environment (?!).
] instantiate; test # Test the code in the environment checked in the manifest. Like you did once. Reproducible.
] update; test # Test the code with latest [compat]ible deps versions. Like your users will shortly do. Non-reproducible.

```

However, I think we do agree on the stakes and the underlying semantics. The disagreement is only about how to leverage these in _practice_. In particular, I agree with you that:

> [@goerz](#):
>
> You want to test the experience a _user_ has when installing the package.

Yes we both do. Only, the way you do this is by issuing `] test` after taking care _not_ to commit the manifest. And the way I do this is by taking care of issuing `] update; test` instead because I _have_ committed the manifest. But we are both sharing the same objective.

Clarifying my position again: I am not suggesting that we _recommend_ people to check in their manifest. I am suggesting that we stop recommending _against_ it. Instead, I think we should document the semantics better and explain, as part of julia docs, what the pitfalls are, what the options are and what are the tradeoffs involved, like the ones we are discussing here.

> [@goerz](#):
>
> [merge conflicts] is probably not a major reason not to commit the Manifest

Agreed. Yet since it may definitely be a source of confusion / frustration among users willing to check it in, then I think it should be officially documented how to handle manifests conflicts.

> [@goerz](#):
>
> I’ve gone down the rabbit hole of custom dev-workflows myself, and it doesn’t scale [..]. However, things have gotten _a lot_ better reently. [..] [`[sources]` feature](https://pkgdocs.julialang.org/v1/toml-files/#The-%5Bsources%5D-section) [..] [`[workspace]` feature](https://pkgdocs.julialang.org/dev/toml-files/#The-%5Bworkspace%5D-section) [..] use as much “standard” tooling with as little customization as possible.

Totally agreed. I also find it very useful that files like [`Manifest-v1.11.toml`](https://pkgdocs.julialang.org/dev/toml-files/#Different-Manifests-for-Different-Julia-versions) exist now. In fact, I think I don’t even need non-standard tooling because my “custom dev-workflow” _can_ be expressed yet in terms of julia `[compat]`, `] update`, `git commit`, `] instantiate; test`, `] update; test`. Nothing really fancy.

> [@goerz](#):
>
> I agree that `Pkg.test` is severely under-documented, and that we should both figure out and document how `Manifest.toml` and `test/Manifest.toml` interact with `Pkg.test`

Good. What shoud we do then? Report against julia docs or Pkg at least?

> [@goerz](#):
>
> However, this does not mean we should encourage people to start committing `Manifest.toml` files.

I don’t think we should _encourage_ it. I think we should _not discourage_. And do a better job at exposing the semantics, stakes, pitfalls and tradeoffs involved, so as to let users choose what suits them best.

> [@goerz](#):
>
> > [@iago-lito](#):
> >
> > Julia being used a lot as a scientific tool, anything improving **reproducibility** is desired, and I think manifest files can help a lot in this respect.
> 
> Manifests for _packages_ have nothing to do with reproducibility. Where you want Manifests is _projects_ [..] That is, the actual research documents _using_ Julia packages.

I have a different opinion on that. If julia packages are used by the research document, then they are also _part_ of this research document. Only they reside in different locations, which the manifest does an excellent job at pointing at.  
On a deeper level: Reproducibility is not only a matter of being able to reproduce a successful analysis/simulation. Reproducibility is also (and arguably _more so_) a matter of being able to _ **address failures** _ in analysis/simulation. Here is an attempt to illustrate my point:  
Suppose that, say, 5 years after having published your “research document”. Someone points out a fishy/smelly detail in your results. You are happy to have committed the manifest of this project, because you can then `] instantiate` it to investigate even though the whole ecosystem has much transformed since. Now, you figure that your project code was actually correct. The problem is that there was a bug in `MyLib`. Now, it is my responsibility to investigate what the bug is, how it was introduced, and attempt to reproduce/fix it. Being a library maintainer yourself, I suppose you can easily picture the **nightmare** it would be had these past ecosystem environments been lost to history. Fortunately, I have checked in all successful `Manifest.toml` builds for `MyLib`. So I can in turn `] instantiate` them and bisect the bug down with really helpful, standard tools.  
To be fair, this is perhaps the only benefit I see of checking in my lib manifests. But I think it is of deep importance and definitely worth the trouble. Plus, I am also arguing that there is no serious trouble.

> [@goerz](#):
>
> Beyond that, I don’t think the benefit of bisectability [..] outweighs the limitation of not testing/developing packages on the same footing as users the package experience.

I think the benefit of bisectability outweights everything for the reason described above. In addition, I don’t think there is such limitation because, IIUC, `] update; test` does exactly that: _“developing on the same footing as users”_, even in the presence of a manifest checked in.  
Again, I figure this is just a matter of taste, practice, and opinion what we mean by ‘reproducibility’ and how much we value that. But if it’s a matter of opinion then, I am arguing that we should stop spreading such strong recommendation _against_ checking the manifest in, especially by invoking Reason 1. Instead, we should document the manifest semantics, and their implications, better.

_ **If** _ you also think so, then I’d actually be happy to summarize this into a draft piece of documentation for review by julia docs or Pkg maintainers, maybe along the lines of what [cargo did](https://blog.rust-lang.org/2023/08/29/committing-lockfiles.html). I am not exactly sure whether/how to do that. Would I just open a PR against their repos? Or ask that they come discuss it here first?

Again, I am happy with the discussion. Thank you for your feedback : )

* * *

[here are other reactions that I didn’t managed to smoothly fit into the above argument]:

> [@goerz](#):
>
> libraries should work with a wide range of versions of their dependencies. So the versions that would be pinned by a manifest do not matter

I think it matters a lot because of my vision of ‘reproducibility’ described above. I think that being able to pin versions is crux to enabling future investigation / debugging in case of late trouble.

> [@goerz](#):
>
> In fact, it’s probably beneficial to “randomly” have different versions of your dependencies during day-to-day development, [..], that mirrors the experience the _user_ of the packages has

Instead of that “randomness” provided by (under-documented) `] test`, I’d rather get some _guarantee_ provided by `] update; test`, exactly for the reason you invoke (mirorring user experience). Plus, this command makes it possible that I ensure future reprodubilicity by checking in my manifest with no harm.

> [@goerz](#):
>
> I usually [explicitly test compatibility with the lowest-version-bound of all dependencies](https://github.com/JuliaDocs/DocumenterCitations.jl/blob/74e2c2cc87a03a30bfd30eb822b10f24b8d1d2cb/.github/workflows/ci.yml#L56)

Yeah! The more I was reading about the matter, the more I came to figure that I was also wanting to do that. IIUC lower-bounding checks is also something that [other projects want to setup](https://github.com/SciML/DifferentialEquations.jl/issues/1080#issuecomment-2762920902). I think it’s maybe a very good practice to spread.  
Are you aware of any “standard tooling” to do that? Has it been discussed for future versions of Julia?

> [@goerz](#):
>
> But that’s something that’s best done by [pinning those versions in CI](https://github.com/JuliaDocs/DocumenterCitations.jl/blob/74e2c2cc87a03a30bfd30eb822b10f24b8d1d2cb/.github/workflows/ci.yml#L71), not by committing a Manifest.

Interesting. Can you elaborate on that please? I figured I could either set up a job that automatically pins all my `[compat]` versions as well, but then keep a safe copy of the corresponding manifest and use _that_ for CI until I eventually need to bump my lower bounds. With the lack of standard tooling for lower-bounding yet, I am interested what you think is the best way to do so, and (of course) why : )

---

<div class="post-metadata">

**Author:** ![iago-lito](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/iago-lito/32/31924_2.png) [@iago-lito](https://discourse.julialang.org/u/iago-lito)\
**Post date:** [March 30, 2025, 11:58am UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/6 "2025-03-30T11:58:01Z")

</div>

[update]: From the discussion [on zulip](https://julialang.zulipchat.com/#narrow/channel/137791-general/topic/Does.20the.20Manifest.20actually.20restricts.20versions.20for.20downstr.2E.2E.2E/with/509017286) (apologies for having spawn two parallel discussions, I’ll try to keep up), I am realizing that my whole argument actually revolves about being able to reproduce the `] test`ing environment. _This_ is the very benefit I see in checking in my `./Manifest.toml`.  
However, checking in my `./test/Manifest.toml` _instead_ would also be very alright, and even _more meaningful_. Unfortunately, the semantics of this file are [officially documented](https://pkgdocs.julialang.org/v1/creating-packages/#Alternative-approach:-test/Project.toml-file-test-specific-dependencies) _not_ to be exactly worked out yet. That is a concern :\

---

<div class="post-metadata">

**Author:** ![greatpet](https://avatars.discourse-cdn.com/v4/letter/g/e495f1/32.png) [@greatpet](https://discourse.julialang.org/u/greatpet)\
**Post date:** [March 30, 2025, 1:02pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/7 "2025-03-30T13:02:00Z")

</div>

My casual suggestion: make `Manifest.toml` available in an optional manner, in a subfolder or alternatively a separate git branch called `rescue`. Normally it won’t be used. But if you ever abandon your library (which we all may do as humans with limited time), someone who desperately needs to use the library five years later can instantiate the last working configuration and possibly start porting the code to work with updated dependencies.

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [March 30, 2025, 3:38pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/8 "2025-03-30T15:38:14Z")

</div>

I think now we’re getting to the root of the issue.

Pinning your test environment, as in, only testing against pinned versions, seems like a bad idea to me (hence: don’t commit Manifest files). But the fundamental _ability_ to reproduce a particular test run, on CI or otherwise, is a good idea.

So yes, the fact that `Pkg` does not specify _exactly_ how a test environment is instantiated in an issue. The [docstring of `Pkg.test`](https://pkgdocs.julialang.org/v1/api/#Pkg.test) is both insufficiently detailed and outdated: it doesn’t even mention `test/Project.toml`.

I don’t think I was really aware that `Pkg.test` takes into account the main `Manifest.toml`. Not that that actually makes any sense: The main `Manifest.toml` file is _not_ for the test environment! It doesn’t include the test dependencies in `[extras]`. If anything, `Pkg.test` should use `test/Manifest.toml` when it exists. I have this vague recollection of `Pkg.test` doing some kind of (unspecified!) “merging” of environments when there are any Manifest files present, but that’s only ever caused errors whenever I encountered it (and the exact behavior may have changed over the years).

This underspecification of how `Pkg.test` actually instantiates the test environment, and the difficulty in getting it to easily test against specific versions of dependency packages has actually made me eschew `Pkg.test` altogether for a long time. Instead, I would use [my own `test()` function](https://github.com/JuliaQuantumControl/QuantumControlTestUtils.jl/blob/6071d156b99f563b15b30ae93784c0c3ddbd6fdc/src/runner.jl#L213) for running `test/runtests.jl` in a subprocess. Alas, with the general improvements in built-in tooling, maintaining such custom workflows is becoming more trouble than it’s worth. For “standalone” packages that can use 1.11 or 1.12 for development tasks, there are “standard” workflows at this point that seem to work well enough.

But yes, `Pkg.test` is still a mess, and we should do better. There should be a way to _exactly_ specify the test environment (presumably, by giving it a full Manifest; none of this “merging environments”). It should also be possible to get a Manifest _out_ of `Pkg.test`. Presumably, a flag to override the temp-folder where `Pkg` instantiates the test project would do the trick.

Once you have the ability to get Manifests in and out of `Pkg.test`, there is still the question of what you do with that. I still wouldn’t _commit_ them. Having them as downloadable artifacts in CI would probably go a long way (although CI logs on GitHub only survive three months, so for long-term archival, you’d have to figure out some archival system).

In any case, yes, you could open issues in `Pkg`. There probably are some already, for example [Document two new ways of specifying test-specific dependencies · Issue #3953 · JuliaLang/Pkg.jl · GitHub](https://github.com/JuliaLang/Pkg.jl/issues/3953). Of course, opening issues doesn’t mean the problem gets solved, so it would probably better to really dive into the code of `Pkg.test` and prepare a pull request that fully documents its actual current behavior. I’ve had a look at that code, though, and it has lots of layers of abstractions, so this is not such an easy task.

---

<div class="post-metadata">

**Author:** ![iago-lito](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/iago-lito/32/31924_2.png) [@iago-lito](https://discourse.julialang.org/u/iago-lito)\
**Post date:** [March 30, 2025, 5:10pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/9 "2025-03-30T17:10:04Z")

</div>

Awesome, I think we’ve drilled down to the gist of it indeed : ) Thank you @goerz \<3

In summary, the reason I want to commit my lib `Manifest.toml` is to _get control over reproducing my `] test` environments_. However, this is not the right way to do so. And there may be no actual “right” way since the current semantics of `] test` are underspecified by `Pkg` itself.

Although I can wait on future features to help me out, I’m obviously afraid I won’t have the bandwidth to upgrade from a consumer of `Pkg` to a contributor and fix that, especially since you write that the task is not easy :\

Nevertheless, I am now agreeing that commiting the a manifest is not semantically the right way to handle that. For one simple reason maybe: should a commit remain the reference for several months, then there could be _several working manifests_ that we would like to record against it as the ecosystem evolves while the code remains fixed. Therefore there is not a 1-1 mapping between the code commits and the project successful manifests eligible for archive.

In the meantime though, I don’t see a better option than to rely on the current behaviour of `Pkg.test` : producing a manifest from the `[compat]` and `[extras]` sections, and commiting it along to obtain a naive record of successful `Pkg.test(; allow_reresolve = false)` environments along my git history. In the situation described above, I would just issue `⬆️ Bump working manifest.` commits here and there as a quick&dirty emulation of the ideal “archival system” you describe. But then I would also become in favour of _not_ recommending this as a typical development worflow.

As you suggest IIUC, there won’t be an ideal workflow until `Pkg.test` solves this :\

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [March 30, 2025, 5:34pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/10 "2025-03-30T17:34:11Z")

</div>

> [@iago-lito](#):
>
> In the meantime though, I don’t see a better option than to rely on the current behaviour of `Pkg.test` : producing a manifest from the `[compat]` and `[extras]` sections

Sounds okay to me in principle, but unless I’m missing something, there isn’t actually a way create a manifest that includes the specifications in `[extras]` (unless maybe some trick exploiting the [`TestEnv` package](https://github.com/JuliaTesting/TestEnv.jl)). You’d be able to keep track of which versions of your package’s direct dependencies you’re testing against, but not which versions of test-only dependencies. So while your approach goes a long way towards your goal, it doesn’t guarantee reproducible test runs.

---

<div class="post-metadata">

**Author:** ![iago-lito](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/iago-lito/32/31924_2.png) [@iago-lito](https://discourse.julialang.org/u/iago-lito)\
**Post date:** [March 30, 2025, 6:41pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/11 "2025-03-30T18:41:50Z")

</div>

Crab, that’s right.

What about dropping `Pkg.test` altogether and issue my own `test/Project.toml` then `julia --project=test test/runtest.jl`?  
I guess I would loose code coverage. And the `test/Project.toml` would not inherit from `Project.toml` anymore so it’d have to duplicate dependencies. Nah, not liking it :\

Maybe with this new [`[workspace]`](https://pkgdocs.julialang.org/dev/toml-files/#The-%5Bworkspace%5D-section) thing?

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [March 30, 2025, 6:49pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/12 "2025-03-30T18:49:52Z")

</div>

> [@iago-lito](#):
>
> What about dropping `Pkg.test` altogether and issue my own `test/Project.toml` then `julia --project=test test/runtest.jl`

Yeah, that’s _exactly_ what I used to do. In my case, with a [lot of custom tooling](https://github.com/JuliaQuantumControl/QuantumControlTestUtils.jl/blob/6071d156b99f563b15b30ae93784c0c3ddbd6fdc/src/runner.jl#L213). Probably more than strictly necessary, since this grew organically for the needs of a whole bunch of unstable packages in the org that need to be developed and tested together. You can definitely write your [`Makefile`](https://github.com/JuliaQuantumControl/QuantumControl.jl/blob/bf7a15e9c9d042ead4e7afa439a875dec631092c/Makefile#L24) and [`CI.yml`](https://github.com/JuliaQuantumControl/QuantumControl.jl/blob/bf7a15e9c9d042ead4e7afa439a875dec631092c/.github/workflows/CI.yml#L36) to execute `test/runtests.jl` directly (with or without coverage), instead of using `Pkg.test` or the [standard actions](https://github.com/julia-actions). But that’s a maintenance burden, and you may end up having problems with, e.g., [PkgEval](https://github.com/JuliaCI/PkgEval.jl) and other systems expecting “default workflows”. In the long term, I’m trying to get away from custom tooling.

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [March 30, 2025, 9:55pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/13 "2025-03-30T21:55:46Z")

</div>

I might have an idea for getting you the kind of reproducibility you’re looking for without any of the drawbacks of committing Manifest files, and within the limitations of the current `Pkg` and standard tooling:

1. Use [TestEnv.jl](https://github.com/JuliaTesting/TestEnv.jl) to get a REPL with the test environment of your package. This should work whether you use `[extras]` or `test/Project.toml`.
2. Use [`Pkg.status()`](https://pkgdocs.julialang.org/v1/api/#Pkg.status) to print the packages and their resolved version numbers in the environment. This would include both explicit and implicit test dependencies
3. Create a file `test/install_supported.jl` where you use [`Pkg.pin`](https://pkgdocs.julialang.org/v1/api/#Pkg.pin) commands to force all packages to the exact version shown earlier by `Pkg.status()`. You could probably write yourself a script to automate this process
4. At the top of `test/run_tests.jl`, conditionally `include` the `install_supported.jl` file. You could make this conditional on environment variables and/or `ARGS`: Both `Pkg.test` and the [`julia-runtest` actions](https://github.com/julia-actions/julia-runtest) allow passing arguments to `Pkg.test`
5. You might consider also creating a file `install_lowest_compat.jl` [where you pin all packages to the minimum version specified in your packages `compat` bounds](https://github.com/JuliaDocs/DocumenterCitations.jl/blob/c1b4f7a2fdf5341b6f57e694c155a5e8104110cd/.github/workflows/ci.yml#L55-L83).
6. In CI, create different jobs as desired, e.g. to run the tests without any arguments (with the “latest” compatible packages, what users would generally experience), with the “supported” versions, and the “lowest compat” versions. Similarly, when testing locally, call `Pkg.test` with the appropriate options the switch between these cases
7. On a regular basis, e.g., before making a release, update the pinned “supported” versions to the latest versions. This is more tedious that using auto-genererated Manifest files, but presumably something that could be automated.

---

<div class="post-metadata">

**Author:** ![iago-lito](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/iago-lito/32/31924_2.png) [@iago-lito](https://discourse.julialang.org/u/iago-lito)\
**Post date:** [March 31, 2025, 7:54am UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/14 "2025-03-31T07:54:14Z")

</div>

Thank you so much for guidance @goerz 💖  
It’ll take some time for me to digest all this. For context, I was still thinking last Monday that versions in my manifest were pinned for downstream consumers (wrong 1) at that it was a good thing to pin them down (wrong 2). I had heard here and there that dependencies handling were kind of suprisingly difficult but I didn’t expect it was such a rabbit hole ^ ^"  
The solution you are offering sounds like an ok compromise between custom / standard workflows, at least for the time / limitations being. I’ll start from it : )

---

<div class="post-metadata">

**Author:** ![j-fu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j-fu/32/11373_2.png) [@j-fu](https://discourse.julialang.org/u/j-fu)\
**Post date:** [March 31, 2025, 10:28am UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/15 "2025-03-31T10:28:36Z")

</div>

I think that abandoning Pkg.test() might harm PkgEval which gives Julia core developers the chance to test the impact of their changes on the ecosystem. In that sense I see it important to improve that.

---

<div class="post-metadata">

**Author:** ![abraemer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/abraemer/32/51403_2.png) [@abraemer](https://discourse.julialang.org/u/abraemer)\
**Post date:** [March 31, 2025, 7:58pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/16 "2025-03-31T19:58:19Z")

</div>

I have been following this thread only loosly so perhaps my idea misses the mark.  
The core concern here is repoducibility, right? This means that we can reproduce the environment tests ran _at some specific point in time_. As all packages keep their history, in principle it would be possible to reconstruct the state of the depencies at some point in time. So perhaps it could be enough to give the package the resolver the option to only consider version that are older than some specific time? Depending on how accessible the time information is to the resolver currently, this might be rather easy to implement. Even it is not accessible currently, I think it shouldn’t be too hard.  
Would this feature be sufficient to achieve the level of reproducibility you folks are looking for? In my mind, the environment would be uniquely defined by something like “use the latest depencies, that where available on June 6th 2023” or similar. So we could keep testing the latest versions by default but can time travel.

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [March 31, 2025, 10:21pm UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/17 "2025-03-31T22:21:47Z")

</div>

Yes, absolutely, that would be a wonderful feature for `Pkg` to have. I’ve certainly been in situations (not just with Julia projects) where a project from a couple of years ago didn’t have the equivalent of a Manifest file. Trying to reproduce projects like that, I’ve definitely taken the approach to go by date: manually create an environment with the versions of all the dependencies that were current at the time the project was originally run. That’s a tedious process, though, so a flag for `Pkg` to say “only consider releases before date X” would be very nice indeed.

---

<div class="post-metadata">

**Author:** ![iago-lito](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/iago-lito/32/31924_2.png) [@iago-lito](https://discourse.julialang.org/u/iago-lito)\
**Post date:** [April 1, 2025, 8:48am UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/18 "2025-04-01T08:48:55Z")

</div>

I would also love that, but worry that it would be fragile to what lib authors do with their commits. If one single lib decides to squash their entire history then the whole process would be doomed, right?

(that being said, I guess that a careful archive of past manifests would not help either)

This actually sounds like a job for [guix](https://guix.gnu.org/en/).

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [April 1, 2025, 9:01am UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/19 "2025-04-01T09:01:04Z")

</div>

> [@iago-lito](#):
>
> If one single lib decides to squash their entire history then the whole process would be doomed, right?

No, this is exactly as robust as a manifest. All versions of registered packages are independently archived by JuliaHub. So even if the original repo was deleted, all Manifests would still be instantiable. Of course, for unregistered packages, all bets are off, in any case. If you’re worried about that, you should archive a snapshot of the library yourself.

This idea for `Pkg` to have a date flag is definitely doable, just someone would have to implement it. That implementation would probably not be trivial, so I wouldn’t hold my breath on this. But who knows. It would definitely be a nice feature.

---

<div class="post-metadata">

**Author:** ![goerz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerz/32/3269_2.png) [@goerz](https://discourse.julialang.org/u/goerz)\
**Post date:** [April 1, 2025, 10:57am UTC](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490/20 "2025-04-01T10:57:48Z")

</div>

Now that I think of it: You could make sure you’re using the uncompressed/cloned version of the General registry (or any other registry), and manually roll it back to some specific point in time.

I’m pretty sure that would result in packages being resolved only using the versions available at that point in time.

To get the uncompressed registry, I think you’d have to do

```julia
pkg> registry rm General # remove compressed version
pkg> registry add https://github.com/JuliaRegistries/General.git
  # add uncompressed version

```

Then go to `$JULIA_DEPOT_PATH/registries/General`, and roll back the registry to some specific point in time. Based on a post from @StefanKarpinski in [Do you have "General.tar.gz" in your `.julia/registries/`? - #5 by StefanKarpinski](https://discourse.julialang.org/t/do-you-have-general-tar-gz-in-your-julia-registries/115234/5), going to a specific historical registry state isn’t just an afterthought.

I’d probably try to play around with this only with clean `JULIA_DEPOT_PATH` (so as not to mess up the default `~/.julia`). I’m also not sure if you need to [take measures to prevent automatic registry updating](https://discourse.julialang.org/t/is-there-a-way-to-prevent-julia-from-updating-the-registries-almost-every-time/64150/30), like `Pkg.offline()`.

Make sure to [delete and re-add `General`](https://pkgdocs.julialang.org/v1/registries/#Adding-registries) when you’re done.

[Next page](https://discourse.julialang.org/t/about-checking-a-package-library-manifest-toml-into-control-version-again/127490.md?page=2)