# Time traveling the dependencies of an older package version

**URL:** <https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874>\
**Category:** Package Management\
**Created:** [December 16, 2024, 8:30am UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874 "2024-12-16T08:30:20Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![MatthijsCox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matthijscox/32/42831_2.png) [@MatthijsCox](https://discourse.julialang.org/u/MatthijsCox)\
**Post date:** [December 16, 2024, 8:30am UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/1 "2024-12-16T08:30:20Z")

</div>

Here’s a package installation problem I occasionally run into. I want to install an older package version, e.g. `PackageA@v0.1`, but when I install the older version it installs newer package dependency versions, say `PackageB@v2` instead of `PackageB@v1`, which are not fully compatible with the desired package `PackageA@v0.1`. So I want to install the dependency versions that were available at the time the package `PackageA@v0.1` was released.

Besides solving compatibility issues, this can also be useful to reproduce the behavior of `PackageA@v0.1` at release time.

I know we could force package authors to be stricter in [compat] usage but it is not possible to change that now in the older package version. So this is not a solution.

Another solution could be that every package stores a Manifest.toml of itself at release time, to have this reproducibility. But again this is not available right now.

Here’s a solution I had in mind:

- check at what time/commit PackageA v0.1 was registered in [General](https://github.com/JuliaRegistries/General)
- at that time/commit in the registry git history, find the available dependency versions
- install only those dependency versions (e.g. by locally cloning General registry of that specific commit and then running `Pkg.add("PackageA")` with it)

I was wondering if someone else had already solved this problem, or whether I should try to write some tool to do the above. Maybe call it `PkgTimeTraveler.jl` or `ManifestMemories.jl`, or something.

---

<div class="post-metadata">

**Author:** ![MatthijsCox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matthijscox/32/42831_2.png) [@MatthijsCox](https://discourse.julialang.org/u/MatthijsCox)\
**Post date:** [December 16, 2024, 8:35am UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/2 "2024-12-16T08:35:02Z")

</div>

Okay just after posting I found another post with the same problem description and a first attempted solution: [Install a package (and its dependencies) at specific date instead of specific version](https://discourse.julialang.org/t/install-a-package-and-its-dependencies-at-specific-date-instead-of-specific-version/91076)

I guess the main challenge is “how to not mess up the main Julia depot” when installing an older version of General registry. I wonder if we could avoid this problem by producing the desired Manifest.toml from the registry history, without actually installing the older registry, and then Pkg.instantiate that with the regular Julia depot/registry.

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [December 16, 2024, 10:11am UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/3 "2024-12-16T10:11:32Z")

</div>

> [@MatthijsCox](#):
>
> I wonder if we could avoid this problem by producing the desired Manifest.toml from the registry history

Interesting idea. If this works, one could really make a package out of it. I wonder, however, if there is an API to querying a registry to produce a `Manifest.toml` without actually instantiating it?

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [December 16, 2024, 10:23am UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/4 "2024-12-16T10:23:06Z")

</div>

> [@MatthijsCox](#):
>
> I guess the main challenge is “how to not mess up the main Julia depot” when installing an older version of General registry.

I don’t think you really mess it up that much, other than you may get some additional package versions you are only temporarily interested in. But if you want to avoid also that, the obvious solution is to use a temporary depot which you can just discard afterwards. I.e. point `JULIA_DEPOT_PATH` to an empty directory and delete that directory afterwards.

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [December 16, 2024, 10:32am UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/5 "2024-12-16T10:32:39Z")

</div>

@GunnarFarneback Do you know whether there is an API in some of (your) registry tools or in Pkg.jl that allows one to create a `Manifest.toml` without actually installing the packages?

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [December 16, 2024, 10:37am UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/6 "2024-12-16T10:37:19Z")

</div>

Not that I know of but Pkg.jl is the place to look since you need to involve the resolver algorithm.

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [December 16, 2024, 10:55am UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/7 "2024-12-16T10:55:38Z")

</div>

> [@MatthijsCox](#):
>
> say `PackageB@v2` instead of `PackageB@v1`, which are not fully compatible with the desired package `PackageA@v0.1`.

Shouldn’t this be prevented by the `[compat]` sections and the version resolver? It would be good to fix the compat ranges in the registry if they are broken for a package.

---

<div class="post-metadata">

**Author:** ![MatthijsCox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matthijscox/32/42831_2.png) [@MatthijsCox](https://discourse.julialang.org/u/MatthijsCox)\
**Post date:** [December 19, 2024, 9:32am UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/8 "2024-12-19T09:32:37Z")

</div>

By reading the Pkg.jl source code, I think I figured out how to only generate the Project.toml and Manifest.toml, without downloading anything:

```julia
using Pkg

function generate_project_env(
    package_name::String,
    registries::Vector{Pkg.Registry.RegistryInstance}=Pkg.Registry.reachable_registries()
)
    pkg = PackageSpec(package_name)
    Pkg.API.handle_package_input!(pkg)
    pkgs = PackageSpec[pkg]
    ctx = Pkg.Operations.Context(registries = registries)
    preserve = Pkg.PRESERVE_NONE

    # found inside Pkg.add, do we need all these?
    # Pkg.add(ctx, pkgs; preserve)
    new_git = Set{Base.UUID}()
    Pkg.Types.project_deps_resolve!(ctx.env, pkgs)
    Pkg.Types.registry_resolve!(ctx.registries, pkgs)
    Pkg.Types.stdlib_resolve!(pkgs)
    Pkg.Types.ensure_resolved(ctx, ctx.env.manifest, pkgs, registry=true)
    #Pkg.API.update_source_if_set(ctx.env.project, pkg)

    # found inside Pkg.Operations.add
    # Pkg.Operations.add(ctx, pkgs, new_git; preserve)
    ctx.env.project.deps[pkg.name] = pkg.uuid
    man_pkgs, deps_map = Pkg.Operations.targeted_resolve(ctx.env, ctx.registries, pkgs, preserve, ctx.julia_version)
    Pkg.Operations.update_manifest!(ctx.env, man_pkgs, deps_map, ctx.julia_version)
    @info "generating Project.toml and Manifest.toml at $(dirname(ctx.env.project_file))"
    Pkg.Operations.write_env(ctx.env)
    return ctx.env
end

Pkg.activate(".")
generate_project_env("JSON")

```

Now I hope I just need to create a `RegistryInstance` for a cloned version of the General registry and pass that along instead of the current registries.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [December 19, 2024, 11:54am UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/9 "2024-12-19T11:54:31Z")

</div>

See an old script of mine that checks out the General registry at arbitrary dates into a temp folder, and uses it for resolving a Julia project: [env\_benchmark.jl · GitHub](https://gist.github.com/aplavin/3f8490c7b944da86f9f1271c8cefaa7e).

---

<div class="post-metadata">

**Author:** ![MatthijsCox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matthijscox/32/42831_2.png) [@MatthijsCox](https://discourse.julialang.org/u/MatthijsCox)\
**Post date:** [December 19, 2024, 2:16pm UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/10 "2024-12-19T14:16:28Z")

</div>

Thanks everyone for the suggestions so far.

Here’s my attempted approach so far, working with large registries like General and other local registries:

1. choose your package and version, e.g. JSON v0.21.0
2. clone _only the git history_ of every reachable registry
  - using `git clone --filter=blob:none --no-checkout --single-branch $registry_url`

3. find which registry your package originally belongs to
4. find the commit and date belonging to the package version
  - assume we can use commit message “New version: JSON v0.21.0”, this is true for General and LocalRegistry registries (ignoring new packages for now)
  - release\_commit = `git rev-list -n 1 --first-parent origin --grep='New version: JSON v0.21.0'`
  - release\_date = `git show -s --format=%ci $release_commit`

5. find the commit of all (other) registries at that date
  - commit = `git rev-list -n 1 --first-parent --before=$release_date origin`

6. clone the commits of the registries, as shallow as possible:
  - `git clone $registry_url --depth=1 $registry_path` (I wish I could skip this somehow)
  - `git fetch --depth=1 origin $commit`
  - `git checkout $commit`

7. create new Julia registry instances based on cloned registries above
  - `Pkg.Registry.RegistryInstance(registry_path)`

8. generate Manifest.toml with previous code
  - `generate_project_env("JSON", cloned_registries)`

I think that should do the trick

---

<div class="post-metadata">

**Author:** ![MatthijsCox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matthijscox/32/42831_2.png) [@MatthijsCox](https://discourse.julialang.org/u/MatthijsCox)\
**Post date:** [December 19, 2024, 2:20pm UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/11 "2024-12-19T14:20:08Z")

</div>

For step 3: to find the corresponding package registry using only the package name I now do the following:

```julia
using Pkg

function find_registry(registries::Vector{Pkg.Registry.RegistryInstance}, pkg_name::String)
    for reg in registries
        uuid_list = Pkg.Registry.uuids_from_name(reg, pkg.name)
        if !isempty(uuid_list)
            return reg
        end
    end
    error("could not find $pkg_name in registries")
end

registries = Pkg.Registry.reachable_registries()
package_registry = find_registry(registries, "JSON")

```

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [December 19, 2024, 4:00pm UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/12 "2024-12-19T16:00:28Z")

</div>

> [@MatthijsCox](#):
>
> clone the commits of the registries, as shallow as possible:
> 
> - `git clone $registry_url --depth=1 $registry_path` (I wish I could skip this somehow)

Assuming all registries are installed as git repos already, you can just copy the `~/.julia/registries/General/` folder (and folders for other registries) to another location and do `git checkout` locally.

---

<div class="post-metadata">

**Author:** ![MatthijsCox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matthijscox/32/42831_2.png) [@MatthijsCox](https://discourse.julialang.org/u/MatthijsCox)\
**Post date:** [December 20, 2024, 10:44am UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/13 "2024-12-20T10:44:42Z")

</div>

> [@aplavin](#):
>
> Assuming all registries are installed as git repos already, you can just copy the `~/.julia/registries/General/` folder (and folders for other registries) to another location and do `git checkout` locally.

I noticed that my General registry last git repo was from 2023. After that it uses the tarball method (since the General registry is just getting too big).

---

<div class="post-metadata">

**Author:** ![MatthijsCox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/matthijscox/32/42831_2.png) [@MatthijsCox](https://discourse.julialang.org/u/MatthijsCox)\
**Post date:** [December 20, 2024, 10:53am UTC](https://discourse.julialang.org/t/time-traveling-the-dependencies-of-an-older-package-version/123874/14 "2024-12-20T10:53:59Z")

</div>

I’ve placed my code inside [RegistryTimeTraveler.jl](https://github.com/matthijscox/RegistryTimeTraveler.jl) for now. It gets the job done for me.

I also found out how to `Pkg.add` with cloned registries in any folder (so no need to change Julia depot) and added that as default example in the readme, instead of using my hacky Manifest.toml generation code.