Add registry from git-tree-sha1 for deterministic Docker builds

Internals & Design
,
1

These days we read a lot about malicious code being introduced to Github.
I am running many julia projects in docker containers, where I want to make sure that the build is done with the same package versions as on my development computer, independent on when I run the pipeline.
However, I am working with different noon-linux machines and package dependencies are not always identical on Mac, Windows and Linux. So I cannot rely on pinning the packages in Manifest.toml.

My approach is fixing the exact version of packages in Project.toml via compat and installing a specific version of the General registry via

function add_registry_from_sha(registry_sha::Union{String, Base.SHA1})
    @info "Installing registry 'General' with git-tree-sha1: $registry_sha"
    depot = joinpath(DEPOT_PATH[1], "registries")

    uuid = "23338594-aafe-5451-b93e-139f81909106"
    mkpath(depot)
    Downloads.download("https://pkg.julialang.org/registry/$uuid/$registry_sha", joinpath(depot, "General.tar.gz"))
    write(joinpath(depot, "General.toml"), """git-tree-sha1 = "$registry_sha"\nuuid = "$uuid"\npath = "General.tar.gz"\n""")
    return nothing
end
  • Wouldn’t it be a good idea to provide this function as
    Pkg.Registry.add(; registry_sha::Union{String, Base.SHA1})?
  • Is there a possibility of preventing the Registry from updating?
2

Packages aren’t allowed to declare OS compat so that Manifests are cross platform by design. Can you explain your situation further?

3

Can’t remember the details but I remember that I received an error when trying to instantiate on a docker from my existing Manifest.toml. Since then I don’t copy Manifest.toml in my Docker setup anymore.

But what you are saying sounds valid, I also don’t see any os-dependency in the resolution mechanism. Maybe I had different julia versions installed and just misinterpreted the error at that time. It’s many years ago and I just didn’t re-think it.

I wonder, whether the basic idea of a pinned registry is nevertheless a good idea. This way one would be free to have the docker running on a different julia version and have the Manifest build with the restriction of a well-defined registry.

EDIT: I just learnt that before julia 1.9 Manifest.toml were platform-dependent. With the new Manifest format 2.0 platform-specific dependencies moved to the packages’ Artifact.toml.