Here’s a package installation problem I occasionally run into. I want to install an older package version, e.g. PackageA@v0.1, but when I install the older version it installs newer package dependency versions, say PackageB@v2 instead of PackageB@v1, which are not fully compatible with the desired package PackageA@v0.1. So I want to install the dependency versions that were available at the time the package PackageA@v0.1 was released.
Besides solving compatibility issues, this can also be useful to reproduce the behavior of PackageA@v0.1 at release time.
I know we could force package authors to be stricter in [compat] usage but it is not possible to change that now in the older package version. So this is not a solution.
Another solution could be that every package stores a Manifest.toml of itself at release time, to have this reproducibility. But again this is not available right now.
Here’s a solution I had in mind:
check at what time/commit PackageA v0.1 was registered in General
at that time/commit in the registry git history, find the available dependency versions
install only those dependency versions (e.g. by locally cloning General registry of that specific commit and then running Pkg.add("PackageA") with it)
I was wondering if someone else had already solved this problem, or whether I should try to write some tool to do the above. Maybe call it PkgTimeTraveler.jl or ManifestMemories.jl, or something.
I guess the main challenge is “how to not mess up the main Julia depot” when installing an older version of General registry. I wonder if we could avoid this problem by producing the desired Manifest.toml from the registry history, without actually installing the older registry, and then Pkg.instantiate that with the regular Julia depot/registry.
Interesting idea. If this works, one could really make a package out of it. I wonder, however, if there is an API to querying a registry to produce a Manifest.toml without actually instantiating it?
I don’t think you really mess it up that much, other than you may get some additional package versions you are only temporarily interested in. But if you want to avoid also that, the obvious solution is to use a temporary depot which you can just discard afterwards. I.e. point JULIA_DEPOT_PATH to an empty directory and delete that directory afterwards.
@GunnarFarneback Do you know whether there is an API in some of (your) registry tools or in Pkg.jl that allows one to create a Manifest.toml without actually installing the packages?
Shouldn’t this be prevented by the [compat] sections and the version resolver? It would be good to fix the compat ranges in the registry if they are broken for a package.
By reading the Pkg.jl source code, I think I figured out how to only generate the Project.toml and Manifest.toml, without downloading anything:
using Pkg
function generate_project_env(
package_name::String,
registries::Vector{Pkg.Registry.RegistryInstance}=Pkg.Registry.reachable_registries()
)
pkg = PackageSpec(package_name)
Pkg.API.handle_package_input!(pkg)
pkgs = PackageSpec[pkg]
ctx = Pkg.Operations.Context(registries = registries)
preserve = Pkg.PRESERVE_NONE
# found inside Pkg.add, do we need all these?
# Pkg.add(ctx, pkgs; preserve)
new_git = Set{Base.UUID}()
Pkg.Types.project_deps_resolve!(ctx.env, pkgs)
Pkg.Types.registry_resolve!(ctx.registries, pkgs)
Pkg.Types.stdlib_resolve!(pkgs)
Pkg.Types.ensure_resolved(ctx, ctx.env.manifest, pkgs, registry=true)
#Pkg.API.update_source_if_set(ctx.env.project, pkg)
# found inside Pkg.Operations.add
# Pkg.Operations.add(ctx, pkgs, new_git; preserve)
ctx.env.project.deps[pkg.name] = pkg.uuid
man_pkgs, deps_map = Pkg.Operations.targeted_resolve(ctx.env, ctx.registries, pkgs, preserve, ctx.julia_version)
Pkg.Operations.update_manifest!(ctx.env, man_pkgs, deps_map, ctx.julia_version)
@info "generating Project.toml and Manifest.toml at $(dirname(ctx.env.project_file))"
Pkg.Operations.write_env(ctx.env)
return ctx.env
end
Pkg.activate(".")
generate_project_env("JSON")
Now I hope I just need to create a RegistryInstance for a cloned version of the General registry and pass that along instead of the current registries.
See an old script of mine that checks out the General registry at arbitrary dates into a temp folder, and uses it for resolving a Julia project: env_benchmark.jl · GitHub.
For step 3: to find the corresponding package registry using only the package name I now do the following:
using Pkg
function find_registry(registries::Vector{Pkg.Registry.RegistryInstance}, pkg_name::String)
for reg in registries
uuid_list = Pkg.Registry.uuids_from_name(reg, pkg.name)
if !isempty(uuid_list)
return reg
end
end
error("could not find $pkg_name in registries")
end
registries = Pkg.Registry.reachable_registries()
package_registry = find_registry(registries, "JSON")
Assuming all registries are installed as git repos already, you can just copy the ~/.julia/registries/General/ folder (and folders for other registries) to another location and do git checkout locally.
I noticed that my General registry last git repo was from 2023. After that it uses the tarball method (since the General registry is just getting too big).
I also found out how to Pkg.add with cloned registries in any folder (so no need to change Julia depot) and added that as default example in the readme, instead of using my hacky Manifest.toml generation code.