About checking a package/library `Manifest.toml` into control version (again)

General Usage
, , , ,
1

Hello @Julia,

From what I understand here and there (but really anywhere), there is almost dogmatic community consensus that Manifest.toml files may be checked into version control for end applications, but certainly not for packages supposed to be consumed library-style by other julia projects downstream.

However, the more I read about the meaning of the manifest, the less I agree or even understand this consensus. I would like to discuss it again.

Although I understand the choice not to track the Manifest.toml (arguably a non-source file), I do not understand that we actually recommend against tracking it. IIUC here are the reasons most frequently invoked in this respect:

Reason 1: Library consumers need maximum flexibility on your dependencies versions so their resolver can construct an environment satisfying all their constraints.

This is totally true, but irrelevant to the topic IMO.
The manifest file is used during an ] instantiate procedure to reproduce the exact same dependencies environment that the one it was created within. However, library consumers are not expected to ] instantiate my project. Instead, they are expected to ] add MyPackage, an action which reads from my Project.toml files, its [deps] section in particular, and then calls into the resolver to satisfy the constraints listed in the [compat] section. IIUC the manifest file is not even actually used during this process.
Therefore, although I think Reason 1 is a good argument against being overly strict in the [compat] section of my project file (e.g. against using strict requirements like "Dep" = "=1.2.3"), I fail to see how it is an argument against checking in my manifest file, which is (again, IIUC) actually ignored by all downstream consumers of my package.

BTW: I may be wrong in this reasoning, but then I’d need to report that how the manifest is used (or not) is not exactly explicit from the official documentation pages: [1] [2]. The above are just the conclusions I came to myself, and confirmed at least there.

Reason 2: Checking in the manifest file yields CI breakage or misleading CI success.

Indeed, FWIU, the following command, typically used in CI:

] test

is actually a synonym of:

] instantiate; test # Build local environment before testing.

Therefore, if my project repo contains a Manifest.toml file, then that file will be used to specify the environment. And the tests will be run in an locked/frozen environment that differs from the one actually used by downstream users supposed to ] add MyLib later, because they would typically get a fresher environment with numerous dependencies bumped.

However, it seems super-easy to avoid that:

] update; test # Bump local environment before testing.

If I check in my manifest file, then I can easily do both:

] instantiate; test # Check that nothing has broken in the code.
] update; test      # Check that nothing has broken in the ecosystem.

Or (if it’s too long):

] update; test # Check both at the same time.

Being able to choose is a clear advantage to having the manifest checked in IMO. Especially because the manifest makes it possible to debug past build and bisect versions history without the ecosystem changes getting in the way.

Reason 3: It creates merge conflicts.

It certainly does.
However, for the same reason you are not supposed to write manifest files as a human, you are not supposed to resolve manifest conflicts as a human.
If you stumble accross an unmerged manifest file, then you must be currently working on development versions of your code. Therefore, I suggest that it is not a problem that you just delete it, make sure you resolve (actual) conflicts within the Project.toml, and then ] update to automatically generate a fresh, “merged” one.

Reason 4: It increases maintenance burden.

I personally see no harm in running ] update; test every so often and committing the resulting changes to my Manifest.toml should the tests still pass. I guess this is also an easy CI automation.

All in all, and unless I am missing something obvious, I would personally decide to keep checking manifests of my julia packages/libraries into control version.

If you also agree with my reasoning, I would maybe further suggest that we start working, as a community, against this unsettling de-facto dogma. First by clarifying the documentation pages in this respect. Then possibly by issuing official recommendations and tradeoff on the same pages.

Julia being used a lot as a scientific tool, anything improving reproducibility is desired, and I think manifest files can help a lot in this respect. We should use them more. Libraries/Packages should not be excluded from this effort.

What do you think? : )

[some more context]:
Almost two years ago, there has been a change in the official recommendations in rust community regarding this very topic. The result is that users, instead of being recommended to not check in their manifest for libraries, are now recommended to do whatever suits their project best. Julia is not rust though. In particular, Julia cannot have two different versions of the same package within the same environment, whereas two different versions of the same crate can coexist within the same rust build. IIUC this makes dependency resolution harder for Julia, which needs to pick only one version for every dependency and yet still satisfy every requirement. However, I think that all the above argument still holds.

1 Like
2

I’m not checking in Manifest.toml files for libraries because libraries should work with a wide range of versions of their dependencies. So the versions that would be pinned by a manifest do not matter, and accordingly, I wouldn’t spent any effort or mental overhead on this. In fact, it’s probably beneficial to “randomly” have different versions of your dependencies during day-to-day development, and specifically to have “latest” versions of the dependencies appear automatically. As a developer, that mirrors the experience the user of the packages has, which seems like a very good idea.

This is not to say that you don’t also sometimes want to test against specific versions of dependencies. For example, I usually explicitly test compatibility with the lowest-version-bound of all dependencies. But that’s something that’s best done by pinning those versions in CI, not by committing a Manifest.

1 Like
3

In more detail,

You are correct that checking in your Manifest.toml has no influence on the users of your package. But that’s not a reason to check in the Manifest.

the manifest is used (or not) is not exactly explicit from the official documentation pages […] Your tests will respect the manifest when resolving versions

Although I would trust @ericphanson’s knowledge on this, that’s completely undocumented (first time I’ve even heard about that), and I’m not sure whether it can be relied upon. Also, I think test dependencies should be specified in test/Projects.toml, not the [extras] section. However, I have no idea how a main Manifest.toml and a test/Manifest.toml would interact (and the docs explicitly say that this is undefined at the moment). I would actually love if Pkg.test would take into account the test/Manifest.toml file, but that has little bearing on whether you should check in that file.

It absolutely produces misleading CI success, and that’s probably the major reason you shouldn’t do this. You want to test the experience a user has when installing the package. That fact that it’s relatively easy to update the Manifest doesn’t mitigate that in any way.

Yeah, this is probably not a major reason not to commit the Manifest

That’s another major reason not to commit the Manifest. I’ve gone down the rabbit hole of custom dev-workflows myself, and it doesn’t scale once you start maintaining more than a very small handful of packages. A lot of these custom workflows were due to my lack of experience, but also a general lack of documentation and tooling a few years ago. However, things have gotten a lot better reently. The [sources] feature in 1.11 is a major step forward, and the [workspace] feature in the upcoming 1.12 seems like be the last piece of the puzzle. With that, my plan would be to use as much “standard” tooling with as little customization as possible.

I agree that Pkg.test is severely under-documented, and that we should both figure out and document how Manifest.toml and test/Manifest.toml interact with Pkg.test. IMO, both of them should interact: Ideally, Pkg.test would use Manifest.toml or test/Manifest.toml according to the same logic it uses to decide where to get the test dependencies.

However, this does not mean we should encourage people to start committing Manifest.toml files.

Julia being used a lot as a scientific tool, anything improving reproducibility is desired, and I think manifest files can help a lot in this respect.

Manifests for packages have nothing to do with reproducibility. Where you want Manifests is projects, like sets of Jupyter/Pluto notebooks or DrWatson-based repos. That is, the actual research documents using Julia packages.

The Rust discussion does raise an interesting point about the reproducibility of CI runs and the potential help with git bisect. It might be nice if the julia-runtest could capture the Manifest.toml file that Pkg used and attach it to the CI run as a artifact. Beyond that, I don’t think the benefit of bisectability with pinned versions outweighs the limitation of not testing/developing packages on the same footing as users the package experience.

1 Like
4

Your tests will respect the manifest when resolving versions.

This is a bit of a subtle point, but technically speaking this is not 100% guaranteed by default.

If you want to guarantee that Pkg will respect the manifest when running your package’s test suite, you need to do Pkg.test(; allow_reresolve=false).

5

Hello @goerz @dilumaluthge and thank you for discussion : )

Indeed. I wouldn’t check it in just because I could ^ ^ I do it because I see benefit in it (see below). At least we agree that “Reason 1” is a misleader. It denotes a misunderstanding what manifests are, which I think possibly reveals under-documentation.

This is.. rather unsettling. Do I understand correctly that the two following commands may not yield the same result?

] instantiate; test # Test within environment pinned by Manifest.toml if this exists.
] test # Test within.. *some* environment, not exactly specified which. Undocumented why.

My gut feeling is that this is worth an issue against julia or julia docs, right? I’d be happy to (at least) report.

Yeah, I have also read this part. TBH I have found this so confusing that I am now actually wanting to revert from our test/Project.toml to some [extras] section instead, just to be sure what julia actually does when I ] test. IIUC you are agreeing that this is slightly off-topic for this thread. But I’m happy to discuss this in another one, or to find good reading on the topic if you happen to have some to share.

I disagree on that. I think it does mitigate. My argument is that if you care about the meaning of CI success, then you are likely a lib maintainer. Therefore, you’ll be easily explained that the following three simple commands have different semantics, and how/whether to use them:

] test # Test the code in some.. undocumented, possibly non-reproducible environment (?!).
] instantiate; test # Test the code in the environment checked in the manifest. Like you did once. Reproducible.
] update; test # Test the code with latest [compat]ible deps versions. Like your users will shortly do. Non-reproducible.

However, I think we do agree on the stakes and the underlying semantics. The disagreement is only about how to leverage these in practice. In particular, I agree with you that:

Yes we both do. Only, the way you do this is by issuing ] test after taking care not to commit the manifest. And the way I do this is by taking care of issuing ] update; test instead because I have committed the manifest. But we are both sharing the same objective.

Clarifying my position again: I am not suggesting that we recommend people to check in their manifest. I am suggesting that we stop recommending against it. Instead, I think we should document the semantics better and explain, as part of julia docs, what the pitfalls are, what the options are and what are the tradeoffs involved, like the ones we are discussing here.

Agreed. Yet since it may definitely be a source of confusion / frustration among users willing to check it in, then I think it should be officially documented how to handle manifests conflicts.

Totally agreed. I also find it very useful that files like Manifest-v1.11.toml exist now. In fact, I think I don’t even need non-standard tooling because my “custom dev-workflow” can be expressed yet in terms of julia [compat], ] update, git commit, ] instantiate; test, ] update; test. Nothing really fancy.

Good. What shoud we do then? Report against julia docs or Pkg at least?

I don’t think we should encourage it. I think we should not discourage. And do a better job at exposing the semantics, stakes, pitfalls and tradeoffs involved, so as to let users choose what suits them best.

I have a different opinion on that. If julia packages are used by the research document, then they are also part of this research document. Only they reside in different locations, which the manifest does an excellent job at pointing at.
On a deeper level: Reproducibility is not only a matter of being able to reproduce a successful analysis/simulation. Reproducibility is also (and arguably more so) a matter of being able to address failures in analysis/simulation. Here is an attempt to illustrate my point:
Suppose that, say, 5 years after having published your “research document”. Someone points out a fishy/smelly detail in your results. You are happy to have committed the manifest of this project, because you can then ] instantiate it to investigate even though the whole ecosystem has much transformed since. Now, you figure that your project code was actually correct. The problem is that there was a bug in MyLib. Now, it is my responsibility to investigate what the bug is, how it was introduced, and attempt to reproduce/fix it. Being a library maintainer yourself, I suppose you can easily picture the nightmare it would be had these past ecosystem environments been lost to history. Fortunately, I have checked in all successful Manifest.toml builds for MyLib. So I can in turn ] instantiate them and bisect the bug down with really helpful, standard tools.
To be fair, this is perhaps the only benefit I see of checking in my lib manifests. But I think it is of deep importance and definitely worth the trouble. Plus, I am also arguing that there is no serious trouble.

I think the benefit of bisectability outweights everything for the reason described above. In addition, I don’t think there is such limitation because, IIUC, ] update; test does exactly that: “developing on the same footing as users”, even in the presence of a manifest checked in.
Again, I figure this is just a matter of taste, practice, and opinion what we mean by ‘reproducibility’ and how much we value that. But if it’s a matter of opinion then, I am arguing that we should stop spreading such strong recommendation against checking the manifest in, especially by invoking Reason 1. Instead, we should document the manifest semantics, and their implications, better.

If you also think so, then I’d actually be happy to summarize this into a draft piece of documentation for review by julia docs or Pkg maintainers, maybe along the lines of what cargo did. I am not exactly sure whether/how to do that. Would I just open a PR against their repos? Or ask that they come discuss it here first?

Again, I am happy with the discussion. Thank you for your feedback : )

[here are other reactions that I didn’t managed to smoothly fit into the above argument]:

I think it matters a lot because of my vision of ‘reproducibility’ described above. I think that being able to pin versions is crux to enabling future investigation / debugging in case of late trouble.

Instead of that “randomness” provided by (under-documented) ] test, I’d rather get some guarantee provided by ] update; test, exactly for the reason you invoke (mirorring user experience). Plus, this command makes it possible that I ensure future reprodubilicity by checking in my manifest with no harm.

Yeah! The more I was reading about the matter, the more I came to figure that I was also wanting to do that. IIUC lower-bounding checks is also something that other projects want to setup. I think it’s maybe a very good practice to spread.
Are you aware of any “standard tooling” to do that? Has it been discussed for future versions of Julia?

Interesting. Can you elaborate on that please? I figured I could either set up a job that automatically pins all my [compat] versions as well, but then keep a safe copy of the corresponding manifest and use that for CI until I eventually need to bump my lower bounds. With the lack of standard tooling for lower-bounding yet, I am interested what you think is the best way to do so, and (of course) why : )

1 Like
6

[update]: From the discussion on zulip (apologies for having spawn two parallel discussions, I’ll try to keep up), I am realizing that my whole argument actually revolves about being able to reproduce the ] testing environment. This is the very benefit I see in checking in my ./Manifest.toml.
However, checking in my ./test/Manifest.toml instead would also be very alright, and even more meaningful. Unfortunately, the semantics of this file are officially documented not to be exactly worked out yet. That is a concern :\

7

My casual suggestion: make Manifest.toml available in an optional manner, in a subfolder or alternatively a separate git branch called rescue. Normally it won’t be used. But if you ever abandon your library (which we all may do as humans with limited time), someone who desperately needs to use the library five years later can instantiate the last working configuration and possibly start porting the code to work with updated dependencies.

8

I think now we’re getting to the root of the issue.

Pinning your test environment, as in, only testing against pinned versions, seems like a bad idea to me (hence: don’t commit Manifest files). But the fundamental ability to reproduce a particular test run, on CI or otherwise, is a good idea.

So yes, the fact that Pkg does not specify exactly how a test environment is instantiated in an issue. The docstring of Pkg.test is both insufficiently detailed and outdated: it doesn’t even mention test/Project.toml.

I don’t think I was really aware that Pkg.test takes into account the main Manifest.toml. Not that that actually makes any sense: The main Manifest.toml file is not for the test environment! It doesn’t include the test dependencies in [extras]. If anything, Pkg.test should use test/Manifest.toml when it exists. I have this vague recollection of Pkg.test doing some kind of (unspecified!) “merging” of environments when there are any Manifest files present, but that’s only ever caused errors whenever I encountered it (and the exact behavior may have changed over the years).

This underspecification of how Pkg.test actually instantiates the test environment, and the difficulty in getting it to easily test against specific versions of dependency packages has actually made me eschew Pkg.test altogether for a long time. Instead, I would use my own test() function for running test/runtests.jl in a subprocess. Alas, with the general improvements in built-in tooling, maintaining such custom workflows is becoming more trouble than it’s worth. For “standalone” packages that can use 1.11 or 1.12 for development tasks, there are “standard” workflows at this point that seem to work well enough.

But yes, Pkg.test is still a mess, and we should do better. There should be a way to exactly specify the test environment (presumably, by giving it a full Manifest; none of this “merging environments”). It should also be possible to get a Manifest out of Pkg.test. Presumably, a flag to override the temp-folder where Pkg instantiates the test project would do the trick.

Once you have the ability to get Manifests in and out of Pkg.test, there is still the question of what you do with that. I still wouldn’t commit them. Having them as downloadable artifacts in CI would probably go a long way (although CI logs on GitHub only survive three months, so for long-term archival, you’d have to figure out some archival system).

In any case, yes, you could open issues in Pkg. There probably are some already, for example Document two new ways of specifying test-specific dependencies · Issue #3953 · JuliaLang/Pkg.jl · GitHub. Of course, opening issues doesn’t mean the problem gets solved, so it would probably better to really dive into the code of Pkg.test and prepare a pull request that fully documents its actual current behavior. I’ve had a look at that code, though, and it has lots of layers of abstractions, so this is not such an easy task.

2 Likes
9

Awesome, I think we’ve drilled down to the gist of it indeed : ) Thank you @goerz <3

In summary, the reason I want to commit my lib Manifest.toml is to get control over reproducing my ] test environments. However, this is not the right way to do so. And there may be no actual “right” way since the current semantics of ] test are underspecified by Pkg itself.

Although I can wait on future features to help me out, I’m obviously afraid I won’t have the bandwidth to upgrade from a consumer of Pkg to a contributor and fix that, especially since you write that the task is not easy :\

Nevertheless, I am now agreeing that commiting the a manifest is not semantically the right way to handle that. For one simple reason maybe: should a commit remain the reference for several months, then there could be several working manifests that we would like to record against it as the ecosystem evolves while the code remains fixed. Therefore there is not a 1-1 mapping between the code commits and the project successful manifests eligible for archive.

In the meantime though, I don’t see a better option than to rely on the current behaviour of Pkg.test : producing a manifest from the [compat] and [extras] sections, and commiting it along to obtain a naive record of successful Pkg.test(; allow_reresolve = false) environments along my git history. In the situation described above, I would just issue ⬆️ Bump working manifest. commits here and there as a quick&dirty emulation of the ideal “archival system” you describe. But then I would also become in favour of not recommending this as a typical development worflow.

As you suggest IIUC, there won’t be an ideal workflow until Pkg.test solves this :\

1 Like
10

Sounds okay to me in principle, but unless I’m missing something, there isn’t actually a way create a manifest that includes the specifications in [extras] (unless maybe some trick exploiting the TestEnv package). You’d be able to keep track of which versions of your package’s direct dependencies you’re testing against, but not which versions of test-only dependencies. So while your approach goes a long way towards your goal, it doesn’t guarantee reproducible test runs.

11

Crab, that’s right.

What about dropping Pkg.test altogether and issue my own test/Project.toml then julia --project=test test/runtest.jl?
I guess I would loose code coverage. And the test/Project.toml would not inherit from Project.toml anymore so it’d have to duplicate dependencies. Nah, not liking it :\

Maybe with this new [workspace] thing?

12

Yeah, that’s exactly what I used to do. In my case, with a lot of custom tooling. Probably more than strictly necessary, since this grew organically for the needs of a whole bunch of unstable packages in the org that need to be developed and tested together. You can definitely write your Makefile and CI.yml to execute test/runtests.jl directly (with or without coverage), instead of using Pkg.test or the standard actions. But that’s a maintenance burden, and you may end up having problems with, e.g., PkgEval and other systems expecting “default workflows”. In the long term, I’m trying to get away from custom tooling.

13

I might have an idea for getting you the kind of reproducibility you’re looking for without any of the drawbacks of committing Manifest files, and within the limitations of the current Pkg and standard tooling:

  1. Use TestEnv.jl to get a REPL with the test environment of your package. This should work whether you use [extras] or test/Project.toml.
  2. Use Pkg.status() to print the packages and their resolved version numbers in the environment. This would include both explicit and implicit test dependencies
  3. Create a file test/install_supported.jl where you use Pkg.pin commands to force all packages to the exact version shown earlier by Pkg.status(). You could probably write yourself a script to automate this process
  4. At the top of test/run_tests.jl, conditionally include the install_supported.jl file. You could make this conditional on environment variables and/or ARGS: Both Pkg.test and the julia-runtest actions allow passing arguments to Pkg.test
  5. You might consider also creating a file install_lowest_compat.jl where you pin all packages to the minimum version specified in your packages compat bounds.
  6. In CI, create different jobs as desired, e.g. to run the tests without any arguments (with the “latest” compatible packages, what users would generally experience), with the “supported” versions, and the “lowest compat” versions. Similarly, when testing locally, call Pkg.test with the appropriate options the switch between these cases
  7. On a regular basis, e.g., before making a release, update the pinned “supported” versions to the latest versions. This is more tedious that using auto-genererated Manifest files, but presumably something that could be automated.
2 Likes
14

Thank you so much for guidance @goerz :sparkling_heart:
It’ll take some time for me to digest all this. For context, I was still thinking last Monday that versions in my manifest were pinned for downstream consumers (wrong 1) at that it was a good thing to pin them down (wrong 2). I had heard here and there that dependencies handling were kind of suprisingly difficult but I didn’t expect it was such a rabbit hole ^ ^"
The solution you are offering sounds like an ok compromise between custom / standard workflows, at least for the time / limitations being. I’ll start from it : )

2 Likes
15

I think that abandoning Pkg.test() might harm PkgEval which gives Julia core developers the chance to test the impact of their changes on the ecosystem. In that sense I see it important to improve that.

1 Like
16

I have been following this thread only loosly so perhaps my idea misses the mark.
The core concern here is repoducibility, right? This means that we can reproduce the environment tests ran at some specific point in time. As all packages keep their history, in principle it would be possible to reconstruct the state of the depencies at some point in time. So perhaps it could be enough to give the package the resolver the option to only consider version that are older than some specific time? Depending on how accessible the time information is to the resolver currently, this might be rather easy to implement. Even it is not accessible currently, I think it shouldn’t be too hard.
Would this feature be sufficient to achieve the level of reproducibility you folks are looking for? In my mind, the environment would be uniquely defined by something like “use the latest depencies, that where available on June 6th 2023” or similar. So we could keep testing the latest versions by default but can time travel.

3 Likes
17

Yes, absolutely, that would be a wonderful feature for Pkg to have. I’ve certainly been in situations (not just with Julia projects) where a project from a couple of years ago didn’t have the equivalent of a Manifest file. Trying to reproduce projects like that, I’ve definitely taken the approach to go by date: manually create an environment with the versions of all the dependencies that were current at the time the project was originally run. That’s a tedious process, though, so a flag for Pkg to say “only consider releases before date X” would be very nice indeed.

1 Like
18

I would also love that, but worry that it would be fragile to what lib authors do with their commits. If one single lib decides to squash their entire history then the whole process would be doomed, right?

(that being said, I guess that a careful archive of past manifests would not help either)

This actually sounds like a job for guix.

19

No, this is exactly as robust as a manifest. All versions of registered packages are independently archived by JuliaHub. So even if the original repo was deleted, all Manifests would still be instantiable. Of course, for unregistered packages, all bets are off, in any case. If you’re worried about that, you should archive a snapshot of the library yourself.

This idea for Pkg to have a date flag is definitely doable, just someone would have to implement it. That implementation would probably not be trivial, so I wouldn’t hold my breath on this. But who knows. It would definitely be a nice feature.

20

Now that I think of it: You could make sure you’re using the uncompressed/cloned version of the General registry (or any other registry), and manually roll it back to some specific point in time.

I’m pretty sure that would result in packages being resolved only using the versions available at that point in time.

To get the uncompressed registry, I think you’d have to do

pkg> registry rm General  # remove compressed version
pkg> registry add https://github.com/JuliaRegistries/General.git
  # add uncompressed version

Then go to $JULIA_DEPOT_PATH/registries/General, and roll back the registry to some specific point in time. Based on a post from @StefanKarpinski in Do you have "General.tar.gz" in your `.julia/registries/`? - #5 by StefanKarpinski, going to a specific historical registry state isn’t just an afterthought.

I’d probably try to play around with this only with clean JULIA_DEPOT_PATH (so as not to mess up the default ~/.julia). I’m also not sure if you need to take measures to prevent automatic registry updating, like Pkg.offline().

Make sure to delete and re-add General when you’re done.

1 Like