We have our manifest.toml in git. Every time we merge there is a conflict, typically on the project_hash. I’m not sure which project_hash is valid (from the HEAD or the current hash). What is the correct policy?
In my repositories I only have the project.toml (with [compat]-entries of course) but always put the Manifest.toml in the gitignore. At least for a repository to a Package, if that is what you are talking about.
yes, well can you explain the justification for your policy to give some insight? Why .gitignore Manifest.toml and why use [compat] in Project.toml?
In the Project.toml the [compat] enries are required to resolve which package versions to use if several packages are depending on the same package, 6. Compatibility · Pkg.jl, see also point 8 of when registering new packages with the Julia registry Automatic merging guidelines · RegistryCI.jl
Then the manifest is auto-generated, and as much as I can I do not git-commit autogenerated files and put them in the git-ignore. The only thing autogenerated files under git lead to is the conflicts you mention.
1 Like
I see. so Manifest.toml can be recreated from Project.toml. [compat] are optional entries that ensure the proper versions of dependencies are installed. Is this correct?
Do you need Mainfest.toml if you want to recreate the exact environment on a computer for reproducing bugs? This is not needed for our project. I get confused on the need for both Project.toml and Manifest.toml.
Thanks
Do you need Manifest.toml if you want to recreate the exact environment on a computer for reproducing bugs?
Exactly.
Another use case is when you want to get back to the exact same dependencies versions.
This why I track Manifest.toml with git for research projects
(where dependencies should be “exact”),
but not for public packages
that are meant to be used by other people, mixed with their own dependencies.
4 Likes
Just for yourself, the [compat] entries are optional but very helpful for Pkg to resolve exactly the state where two packages depend on the same third package (that is which version this third package to install).
If you register a package in the Julia Registry, compat entries are mandatory for all your dependencies.
Yes the Manifest can be auptmaticaly recreated from the Project. toml, but it might vary slightly in the following sense:
If you have your now Package and depend on say some package in version 0.3.24 and the compat is 0.3.22 (that is all following o.3.22 are allowed before reaching 0.4 and that package releases 0.3.25) a new manifest on another machine will use 0.3.25 instead of 0.3.24. But that should be non-breaking.
So to that extend you loose reproducibility if you do not keep the manifest, tough that should only affect non-breaking changes, so the code will (± bugs) still be fine.
That is to say: For the results of a paper (but not for a Julia Package) I would also commit the Manifest for the final variant of the submitted paper, but also only update this in revised version commits and leave both the Project and Manifest untouched inbetween and not even to an ] update on that environment.
2 Likes
thanks this was really helpful; it is nearly our use case.
1 Like
I have one more question. Suppose that we keep manifest.toml in git. how do you resolve the merge conflicts? E.g., with the project hash?
I don’t know what “project hash” refers to in this context but basically you have two modes of working.
- You care about exact reproducibility. Then you check in
Manifest.tomland arrange your workflows so you don’t do any local modifications to the Manifest and thus no merge conflicts with the Manifest. - You don’t care about exact reproducibility and don’t check in
Manifest.toml, thus not getting merge conflicts.
Not caring about reproducibility but checking in Manifest.toml is just making life difficult for yourself. If I had to work under those conditions and ran into a merge conflict I would just delete the Manifest and do a Pkg.resolve.
1 Like
There is a project hash at the top of the manifest.toml, which is auto-generated.
I think the use case that I have in mind is not really supported by Manifest.toml, which is to be able to add and remove packages, but the packages that remain must all have fixed versions for the duration of the project. It seems that this might be possible through [compat] in project.toml.
It sounds like Manifest.toml is to be used when the dependencies are completely determined and the environment needs to be replicated. Is that interpretation correct?
“completely determined” can be done by also submitting the manifest, sure, but as you noticed with the project hash – that might also lead to merge conflicts usually.
You can fix the versions of your (direct) dependencies also in the [compat] section of the Project.toml, see 6. Compatibility · Pkg.jl and all other specifiers on that page are worth a read as well 