We have our manifest.toml in git. Every time we merge there is a conflict, typically on the project_hash. I’m not sure which project_hash is valid (from the HEAD or the current hash). What is the correct policy?
In my repositories I only have the project.toml (with [compat]-entries of course) but always put the Manifest.toml in the gitignore. At least for a repository to a Package, if that is what you are talking about.
yes, well can you explain the justification for your policy to give some insight? Why .gitignore Manifest.toml and why use [compat] in Project.toml?
In the Project.toml the [compat] enries are required to resolve which package versions to use if several packages are depending on the same package, 6. Compatibility · Pkg.jl, see also point 8 of when registering new packages with the Julia registry Automatic merging guidelines · RegistryCI.jl
Then the manifest is auto-generated, and as much as I can I do not git-commit autogenerated files and put them in the git-ignore. The only thing autogenerated files under git lead to is the conflicts you mention.
1 Like
I see. so Manifest.toml can be recreated from Project.toml. [compat] are optional entries that ensure the proper versions of dependencies are installed. Is this correct?
Do you need Mainfest.toml if you want to recreate the exact environment on a computer for reproducing bugs? This is not needed for our project. I get confused on the need for both Project.toml and Manifest.toml.
Thanks
Do you need Manifest.toml if you want to recreate the exact environment on a computer for reproducing bugs?
Exactly.
Another use case is when you want to get back to the exact same dependencies versions.
This why I track Manifest.toml with git for research projects
(where dependencies should be “exact”),
but not for public packages
that are meant to be used by other people, mixed with their own dependencies.
4 Likes
Just for yourself, the [compat] entries are optional but very helpful for Pkg to resolve exactly the state where two packages depend on the same third package (that is which version this third package to install).
If you register a package in the Julia Registry, compat entries are mandatory for all your dependencies.
Yes the Manifest can be auptmaticaly recreated from the Project. toml, but it might vary slightly in the following sense:
If you have your now Package and depend on say some package in version 0.3.24 and the compat is 0.3.22 (that is all following o.3.22 are allowed before reaching 0.4 and that package releases 0.3.25) a new manifest on another machine will use 0.3.25 instead of 0.3.24. But that should be non-breaking.
So to that extend you loose reproducibility if you do not keep the manifest, tough that should only affect non-breaking changes, so the code will (± bugs) still be fine.
That is to say: For the results of a paper (but not for a Julia Package) I would also commit the Manifest for the final variant of the submitted paper, but also only update this in revised version commits and leave both the Project and Manifest untouched inbetween and not even to an ] update on that environment.
3 Likes
thanks this was really helpful; it is nearly our use case.
1 Like
I have one more question. Suppose that we keep manifest.toml in git. how do you resolve the merge conflicts? E.g., with the project hash?
I don’t know what “project hash” refers to in this context but basically you have two modes of working.
- You care about exact reproducibility. Then you check in
Manifest.tomland arrange your workflows so you don’t do any local modifications to the Manifest and thus no merge conflicts with the Manifest. - You don’t care about exact reproducibility and don’t check in
Manifest.toml, thus not getting merge conflicts.
Not caring about reproducibility but checking in Manifest.toml is just making life difficult for yourself. If I had to work under those conditions and ran into a merge conflict I would just delete the Manifest and do a Pkg.resolve.
2 Likes
There is a project hash at the top of the manifest.toml, which is auto-generated.
I think the use case that I have in mind is not really supported by Manifest.toml, which is to be able to add and remove packages, but the packages that remain must all have fixed versions for the duration of the project. It seems that this might be possible through [compat] in project.toml.
It sounds like Manifest.toml is to be used when the dependencies are completely determined and the environment needs to be replicated. Is that interpretation correct?
“completely determined” can be done by also submitting the manifest, sure, but as you noticed with the project hash – that might also lead to merge conflicts usually.
You can fix the versions of your (direct) dependencies also in the [compat] section of the Project.toml, see 6. Compatibility · Pkg.jl and all other specifiers on that page are worth a read as well 
I see this thread was never resolved, but we have the same problem as well.
For an application, rather than a package, committing the Manifest.toml is the norm. See this page for the distinction between applications and packages: 9. Glossary · Pkg.jl.
Currently, we get these git conflicts all the time, and our devs just have to manually resolve them.
Is there, or could there be, any way to disable the package_hash? @kristoffer.carlsson what value does it provide for a case like this?
2 Likes
The project hash is there to see if there has been changes made to the project file that has not yet been resolved so that a corresponding (updated) manifest file has been generated.
It is unfortunate that it generates conflicts, I guess this is because different devs have different versions of the project file checked out that they are working with when creating PRs.
Sure, there could just be an option to disable it all together but I wonder if there is some better solution… Maybe not…
In my experience a typical scenario is that two feature branches are created from the same commit and then needs to add different new dependencies. When they are both merged back and if you are sufficiently lucky, both Project.toml and the Manifest.toml, apart from the project hash, may merge cleanly. However, there is no way for git to merge the project hashes, even if the Manifest content has been merged to match a resolution of the merged Project files.
However, the more common case is that you aren’t lucky and the Manifest updates diverge in resolved versions, either because of differences in Compat in the new dependencies or because some common dependencies have, possibly unnecessarily, been updated at different times to new versions. In this case a manual merge is basically impossible and you have to resort to a fresh resolve of the merged Project file anyway.
1 Like
Yes, exactly, what Gunnar said. They could be two valid changes, both forked off the same main branch commit, and then they will have a merge conflict when merging back to main.
The project hash is there to see if there has been changes made to the project file that has not yet been resolved so that a corresponding (updated) manifest file has been generated.
Could we maybe move the hash to a separate file that isn’t committed? I think that would address our problems. Or find some git-friendlier format for the hash, such as sharding different parts of the file into different hashes to reduce the likelihood of conflict or something?