Manifest.toml for reproducible code: is it even useful cross-platform?

The Manifest.toml records the state of all installed packages of a “project” (I’m interested in “projects” here not “packages”). I’m collaborating on a project with someone on Windows, I’m on Linux. In general, instantiating the project does not work on Linux with the manifest from Windows, I think that is due to platform specific packages, such as GitHub - JuliaBinaryWrappers/Wayland_jll.jl.

Questions:

  • is there a point of version-controlling the Manifest.toml? (this related discussion seems to say, “yes” for projects) If so, how do I prevent problems of the Manifest not working with instantiate?
  • if no Manifest.toml, what are the ways to keep things as reproducible as possible? This specific case is for code doing data-processing for a research-publication, which I’d like to submit with the publication.

What’s the problem exactly? Package resolution isn’t platform-dependent, as far as I know.

One thing to always keep in mind is that generally speaking a manifest generated with Julia vX.N may not be instantiated with Julia vX.M, with M < N (this is due to the fact standard libraries may change between minor versions of Julia and Pkg doesn’t know how to resolve standard libraries which don’t exist in the current version of Julia). So for example if you’re trying to instantiate with Julia v1.5 a manifest generated with Julia v1.6 you’re likely going to have troubles (dozens of standard libraries have been introduced in v1.6), regardless of the platforms involved.

2 Likes

The Manifest.toml only guarantees that, provided the given dependencies are available for a given platform, you’ll get the same versions of those packages. In particular, Manifest.toml does not give you a portability guarantee across operating systems in case some of those dependencies are OS specific.

Yes, if you want to provide the exact versions of packages you used in your development experience. This does not mean that they will necessarily be reproducible on every platform out there (e.g. if some dependencies are only available on Linux, but not Windows. In those cases you’d usually try to avoid those dependencies, if you want to keep them portable).

That depends on your specific situation - for one, I’m questioning why data-procesing requires a hard dependency on a very linux specific window layouting server?

1 Like

Thanks! Yes, I think the error I did see this time around, was due to using the “wrong” Julia version. If I recall correctly, there are plans to include the Julia version within the Manifest.toml and/or Project.toml, right?

The Manifest.toml only guarantees that, provided the given dependencies are available for a given platform, you’ll get the same versions of those packages. In particular, Manifest.toml does not give you a portability guarantee across operating systems in case some of those dependencies are OS specific.

What happens if that is the case (say Linux Manifest trying to be installed on Windows)? Will an error be thrown or will the necessary deps be installed (and the incompatible ones be ignored)? Or how could they be installed, without upgrading everything?

That depends on your specific situation - for one, I’m questioning why data-procesing requires a hard dependency on a very linux specific window layouting server?

I suspect from the plotting packages. It’s good to have the scripts which produce the paper’s figures within the code.

I think the Julia version is already part of the deps in Project.toml?

The Julia version is only in Project.toml files for packages. If you just activate . in a folder to make a new project it will not put the Julia version in the file.

1 Like

Sorry, I meant compat.

Yea I knew what you meant, still not there. That is only for packages.

I activated a new project in a folder that didn’t have one existing, and I added DataFrames.jl:

(@v1.6) pkg> activate .
  Activating new environment at `/hdd/Data/TradingAlgos/Project.toml`

(TradingAlgos) pkg> add DataFrames
    Updating registry at `~/.julia/registries/General`

This is the complete Project.toml file:

tyler@pop-os:/hdd/Data/TradingAlgos$ cat Project.toml 
[deps]
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"

Again, the package resolver is platform-independent, so I’m not entirely sure what you’re talking about. If you refer to the JLL packages, if there are no artifacts for a specific platform (like Wayland on Windows), nothing happens, no error is thrown, and the corresponding JLL package (e.g. Wayland_jll) is almost no-op.

4 Likes

Ok, then I think I was assuming that errors from using wrong Julia versions were due to Linux-Windows incompatibilities. My bad. Thanks for clearing this up.

I didn’t mean that the julia executable would appear there by magic. I meant that I added it by hand.

Edit: That julia version is specified in the project compat, but it appears to be ignored. I just checked.

That sounds like a bug…