One of the things I find very appealing about julia is the ability to provide, through {Project,Manifest}.toml files, the exact environment some code was run in so enabling somebody else to reproduce the results generated.
To save me looking like a foolish fanboy is this something that is unique to julia? I’m not familiar enough other languages to know if they have this idea too.
Many thanks.
There are ways to do this in principle with other languages; for example with a conda environment, you can run a command to spit out all of the dependencies and exact versions of packages, then use the output to create a new environment that should be identical. But in practice, it’s way more clunky and way less robust than julia’s approach in my experience.
Also, if you’re interested in this feature for doing science, check out DrWatson.jl
A disadvantage of conda is that if you create an environment.yaml file with fixed dependencies / versions under Windows, it will often not work under Linux and vice-versa, because some dependencies of popular packages are different.
I have a list of these longer than my arm 
AFAIK Rust/Cargo also has a focus on reproducibility, and their experience motivated some of the design of the current Pkg system.
I think that taking reproducibility seriously is a relatively recent phenomenon, but eventually most widely used languages will move in this direction: the benefits are tangible, and there are no long-run downsides (of course someone has to implement it, and there is a transition cost).
One thing that separates the way Julia does this from most other systems is that it is reproducible by default: if you use Julia the normal way, you have a manifest that can be version controlled and since git will track it unless you tell it not to, tracking it is also the default. Other package managers often have some way to record precise versions of packages, but in Julia adding dependencies to the manifest is necessary to load the code in the first place, so the manifest is always there and always up to date.
Another notable feature of the system is its portability. Partially this is actually the absence of features: we have stubbornly refused to add any features that allow manifests to be platform-specific. This initially caused some annoyance, but these days the incredible BinaryBuilder system gives us pre-built, cross-compiled binaries and the artifacts system allows developers to easily use those pre-built binaries on all platforms easily and consistently.
A subtle but significant feature of how BinaryBuilder works that makes it more reproducible across platforms than other systems is that it cross-compiles everything for all platforms from a common compiler environment. This means that the binaries for, e.g. Linux x86-64 and Windows x86-64, contain largely same machine code since it was produced by the same compiler—the only difference is how the compiler wraps that machine code up in a binary format for the operating system. Since the actual machine code is the same, libraries are much more likely to work the same than if they had been generated, e.g. by GCC on Linux vs. MSCV on Windows, which is how other systems typically do this.
In summary, since manifests and artifact files are version controlled along with project source, Julia projects are reproducible down to Julia and binary dependencies. Not only that, but this is also portable: you can check a project out on a different platform and as long as there are binaries built for that platform, it should work the same. I’d be interested in hearing how portable Cargo is in this respect, but that’s the only other native system (i.e. not managed runtimes like JVM or CLR) I’m aware of that I’m aware of that offers comparable levels of reproducibility and portability.
% this. I’ve done enough sysadin work to know how wonderful it is to have binary dependencies that actually work. Compiling dependencies is the least fun thing around.
Not always. Thinking the environment tomls are always enough for reproducibility keeps biting me in the butt. The environments stack, so doing things like the following works.
using Pkg
Pkg.activate()
Pkg.add("BenchmarkTools")
Pkg.activate("NewEnv")
Pkg.status() # empty environment
using BenchmarkTools
If you were to share the NewEnv tomls, you would not be able to run the 2nd code block. I find myself prototyping using the default env and then move to a project specific env when things get more concrete. Because of this, I find that I will often pass tests locally (only because of the env stacking), and will find the issue only when CI fails due to an incomplete TOML.
I’m not quite sure how to change my workflow to make the reproducibility more “fool-proof”.
Related to BinaryBuilder, there isn’t yet the goal to have reproducible binaries, but we already have in place some strategies toward that goal. And having a single build environment with predictable paths makes some things easier.
The load path only affects what you can load from the Main module (top-level code). This means it’s possible to write a script that isn’t reproducible but anything that’s wrapped up in proper project code is reproducible. It might be good to be stricter about this, but that has to be balanced against making Julia REPL and script programming too annoying.
This makes sense. I recently hit this issue but it was with scripts. However, I feel like I’ve had PR’s to Quadrature.jl that hit this exact issue. I was able to pass tests locally, but not in GitHub CI due to an incomplete toml. It was a few months ago, so I may be remembering incorrectly.
Not sure how that would happen and it shouldn’t. It would be great if you could file an issue on Pkg if it happens.
Will do. I’m trying to dig through the various PRs to see if I can find the issue. It may have been with respect to a test extra.
I have experienced the same problem. Is there a version of ]activate that replaces e.g. v1.5 as opposed to pushing another environment on the stack?