I created a package that makes reproducable research easier. It provides the functions freeze(Pkg) and lower_bound(), that adds the current versions of your environment to your Project.toml file.
See: PkgHelpers
It comes in two flavours, it can do a full freeze, or a relaxed freeze where updates of patch versions are still allowed.
Just run freeze() before you submit your paper, and even in many years people will be able to reproduce your results.
Installation:
For now you must do, preferably in your global environment:
using Pkg
Pkg.add("https://github.com/ufechner7/PkgHelpers.jl")
But the package is being registered, so in a few days:
Just to make sure I understand. freeze() will add the exact current exact versions of all dependencies (also indirect, from Manifest.toml) to the [compat] section of Project.toml?
Isn’t this equivalent to simply committing your Manifest.toml and then telling people to instantiate before running your code?
OTH, something I would find very useful, is a function that fills [compat] “automatically”, by adding the current versions of all packages (direct dependencies only, so those already listed in Project.toml) with “caret specifiers”, so lower-bounding the compat:
Well, it will only add the direct dependencies, not the indirect dependencies…
My experience with adding Manifest.toml to git is bad, you often get merge conflicts if you use different computers to work on the same project. With this approach it all works well, different computers, different OS, different Julia version, all fine…
For timescales of years, Manifest.toml (running in the same julia version) is really the best approach. It’s the official and battle-tested solution.
The “freeze” name could also be misleading, because it doesn’t freeze all dependencies – only direct ones.
Well, it never worked for me. Renaming Manifest.toml to something like Manifest.toml.bak for documentation purposes might be OK, but putting Manifest.toml into git directly always causes trouble. Don’t do it.
nice.
I proposed this as a potential interface for Pkg to use to control package versions,
Get things to a good version then call freeze, and caret bound everything there.
Its great to see someone else had the same idea and implemented it.
Because you easily get merge errors that are very hard to solve. Well, if you work on more than one computer on a project. I have two laptops and a desktop with different operating systems, and sometimes also different Julia versions.
Committing the project file does not cause any issue, if there should be merge error it is very easy to solve manually (usually only one or two lines differ).
Well, this doesn’t work in practice. When are finished with a project? When you handed in the draft of the paper? When you handed in the final version? When you addressed the comments of the reviewers?
This just doesn’t work. But using my function freeze(), you can do that every month without any issue… and if you have nothing else to do and want to try new package versions un-freeze you project, update, check if everything still works and freeze it again or revert the changes…
I still think that 99% of the time you do not need the Manifest.toml, but of course, if things go wrong, you might need it.
But with a Project.toml with freezed package versions, if in 5 years (I will be pensioner by then) a new researcher tries to pick up my work she or he can use the latest 1.xx Julia version and it will most likely still work…
If it doesn’t, having a backup of the Manifest.toml file and being forced to use a very old Julia version could still be a backup solution…
I know what you mean. What I’ve found when I’ve encountered diverging manifests is that I am in that situation never particularly attached to either manifest and in every case the correct action has been to just delete the conflicted manifest and Pkg.resolve to get a new one.
Yes, it would be nice if git could automate it for me, but it’s not that much of a hurdle.
My experience is that using a project with a committed manifest absolutely does work. Yes, you can’t mix Julia versions but sharing it between operating systems hasn’t been an issue.
In the end it comes down to a tradeoff between how strong reproducibility you want and how much inconvenience you can tolerate. For my use cases it’s not worth the risk of changes in the results because some transitive dependency fixed a bug or introduced a new bug, so I always commit the manifest in such projects. For other use cases exact reproducibility may be less important than being able to mix Julia versions, etc.