# Experimental reproducibility: julia vs the rest

**URL:** <https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769>\
**Category:** General Usage\
**Tags:** reproducibility\
**Created:** [September 17, 2020, 10:59am UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769 "2020-09-17T10:59:17Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![healyp](https://avatars.discourse-cdn.com/v4/letter/h/67e7ee/32.png) [@healyp](https://discourse.julialang.org/u/healyp)\
**Post date:** [September 17, 2020, 10:59am UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/1 "2020-09-17T10:59:17Z")

</div>

One of the things I find very appealing about julia is the ability to provide, through `{Project,Manifest}.toml` files, the exact environment some code was run in so enabling somebody else to reproduce the results generated.

To save me looking like a foolish fanboy is this something that is unique to julia? I’m not familiar enough other languages to know if they have this idea too.

Many thanks.

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [September 17, 2020, 11:22am UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/2 "2020-09-17T11:22:44Z")

</div>

There are ways to do this in principle with other languages; for example with a conda environment, you can run a command to spit out all of the dependencies and exact versions of packages, then use the output to create a new environment that should be identical. But in practice, it’s way more clunky and way less robust than julia’s approach in my experience.

Also, if you’re interested in this feature for doing science, check out DrWatson.jl

---

<div class="post-metadata">

**Author:** ![lungben](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lungben/32/12314_2.png) [@lungben](https://discourse.julialang.org/u/lungben)\
**Post date:** [September 17, 2020, 12:19pm UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/3 "2020-09-17T12:19:58Z")

</div>

A disadvantage of conda is that if you create an `environment.yaml` file with fixed dependencies / versions under Windows, it will often not work under Linux and vice-versa, because some dependencies of popular packages are different.

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [September 17, 2020, 12:22pm UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/4 "2020-09-17T12:22:56Z")

</div>

> [@lungben](#):
>
> A disadvantage of conda

I have a list of these longer than my arm 😂

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 17, 2020, 3:47pm UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/5 "2020-09-17T15:47:45Z")

</div>

AFAIK Rust/Cargo also has a focus on reproducibility, and their experience motivated some of the design of the current Pkg system.

I think that taking reproducibility seriously is a relatively recent phenomenon, but eventually most widely used languages will move in this direction: the benefits are tangible, and there are no long-run downsides (of course someone has to implement it, and there is a transition cost).

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [September 17, 2020, 4:48pm UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/6 "2020-09-17T16:48:34Z")

</div>

One thing that separates the way Julia does this from most other systems is that it is reproducible **by default** : if you use Julia the normal way, you have a manifest that can be version controlled and since git will track it unless you tell it not to, tracking it is also the default. Other package managers often have some way to record precise versions of packages, but in Julia adding dependencies to the manifest is necessary to load the code in the first place, so the manifest is always there and always up to date.

Another notable feature of the system is its portability. Partially this is actually the absence of features: we have stubbornly refused to add any features that allow manifests to be platform-specific. This initially caused some annoyance, but these days the incredible [BinaryBuilder](https://binarybuilder.org) system gives us pre-built, cross-compiled binaries and the [artifacts](https://julialang.github.io/Pkg.jl/v1/artifacts/) system allows developers to easily use those pre-built binaries on all platforms easily and consistently.

A subtle but significant feature of how BinaryBuilder works that makes it more reproducible across platforms than other systems is that it cross-compiles everything for all platforms from a common compiler environment. This means that the binaries for, e.g. Linux x86-64 and Windows x86-64, contain largely same machine code since it was produced by the same compiler—the only difference is how the compiler wraps that machine code up in a binary format for the operating system. Since the actual machine code is the same, libraries are much more likely to work the same than if they had been generated, e.g. by GCC on Linux vs. MSCV on Windows, which is how other systems typically do this.

In summary, since manifests and artifact files are version controlled along with project source, Julia projects are reproducible down to Julia and binary dependencies. Not only that, but this is also portable: you can check a project out on a different platform and as long as there are binaries built for that platform, it should work the same. I’d be interested in hearing how portable Cargo is in this respect, but that’s the only other native system (i.e. not managed runtimes like JVM or CLR) I’m aware of that I’m aware of that offers comparable levels of reproducibility and portability.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [September 17, 2020, 4:54pm UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/7 "2020-09-17T16:54:07Z")

</div>

💯% this. I’ve done enough sysadin work to know how wonderful it is to have binary dependencies that actually work. Compiling dependencies is the least fun thing around.

---

<div class="post-metadata">

**Author:** ![DrPapa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drpapa/32/6835_2.png) [@DrPapa](https://discourse.julialang.org/u/DrPapa)\
**Post date:** [September 17, 2020, 5:04pm UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/8 "2020-09-17T17:04:29Z")

</div>

> [@healyp](#):
>
> through `{Project,Manifest}.toml` files, the exact environment some code was run in so enabling somebody else to reproduce the results generated.

Not always. Thinking the environment tomls are always enough for reproducibility keeps biting me in the butt. The environments stack, so doing things like the following works.

```julia
using Pkg
Pkg.activate()
Pkg.add("BenchmarkTools")

```

```julia
Pkg.activate("NewEnv")
Pkg.status() # empty environment

using BenchmarkTools

```

If you were to share the `NewEnv` tomls, you would not be able to run the 2nd code block. I find myself prototyping using the default env and then move to a project specific env when things get more concrete. Because of this, I find that I will often pass tests locally (only because of the env stacking), and will find the issue only when CI fails due to an incomplete TOML.

I’m not quite sure how to change my workflow to make the reproducibility more “fool-proof”.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [September 17, 2020, 5:12pm UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/9 "2020-09-17T17:12:51Z")

</div>

Related to BinaryBuilder, there isn’t yet the goal to have [reproducible binaries](https://reproducible-builds.org/), but we already have in place some strategies toward that goal. And having a single build environment with predictable paths makes some things easier.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [September 17, 2020, 5:12pm UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/10 "2020-09-17T17:12:54Z")

</div>

The load path only affects what you can load from the Main module (top-level code). This means it’s possible to write a script that isn’t reproducible but anything that’s wrapped up in proper project code is reproducible. It might be good to be stricter about this, but that has to be balanced against making Julia REPL and script programming too annoying.

---

<div class="post-metadata">

**Author:** ![DrPapa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drpapa/32/6835_2.png) [@DrPapa](https://discourse.julialang.org/u/DrPapa)\
**Post date:** [September 17, 2020, 5:26pm UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/11 "2020-09-17T17:26:26Z")

</div>

> [@StefanKarpinski](#):
>
> This means it’s possible to write a script that isn’t reproducible but anything that’s wrapped up in proper project code is reproducible

This makes sense. I recently hit this issue but it was with scripts. However, I feel like I’ve had PR’s to Quadrature.jl that hit this exact issue. I was able to pass tests locally, but not in GitHub CI due to an incomplete toml. It was a few months ago, so I may be remembering incorrectly.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [September 17, 2020, 5:34pm UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/12 "2020-09-17T17:34:32Z")

</div>

Not sure how that would happen and it shouldn’t. It would be great if you could file an issue on [Pkg](https://github.com/JuliaLang/Pkg.jl) if it happens.

---

<div class="post-metadata">

**Author:** ![DrPapa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drpapa/32/6835_2.png) [@DrPapa](https://discourse.julialang.org/u/DrPapa)\
**Post date:** [September 17, 2020, 5:40pm UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/13 "2020-09-17T17:40:09Z")

</div>

Will do. I’m trying to dig through the various PRs to see if I can find the issue. It may have been with respect to a test extra.

---

<div class="post-metadata">

**Author:** ![krcools](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/krcools/32/10212_2.png) [@krcools](https://discourse.julialang.org/u/krcools)\
**Post date:** [September 23, 2020, 12:40pm UTC](https://discourse.julialang.org/t/experimental-reproducibility-julia-vs-the-rest/46769/14 "2020-09-23T12:40:27Z")

</div>

I have experienced the same problem. Is there a version of `]activate` that replaces e.g. `v1.5` as opposed to pushing another environment on the stack?