Compatibility specification with version number and package location?

General Usage
,
1

If I’m developing PkgA and have a dependency to a package that I’ve forked from github (and exists in the general registry) and I add this package to my Project.toml with

pkg>add http://github.com/ForkedPackage.jl

How can I specify the compatibility that PkgA must use this specific package at the repo address?

2

I may be wrong, but I believe this kind of perfect reproducibility (especially bypassing the default registry) is responsibility of the Manifest.toml not the Project.toml.

3

Wouldn’t it be possible to specify the compat bounds exactly?

4

The compat bounds in Project.toml only relate to registry information (*). If you want to specify a custom version by this mechanism you have to replace or complement the General registry with your own registry.

Normally you would just rely on and, if necessary distribute, Manifest.toml for PkgA, which can record all details of how you have added/developed its dependencies. The main limitation of this is that it only helps when working with PkgA itself.

If you in turn need PkgA as dependency of something else things become more complicated. Either you can explicitly add your ForkedPackage there as well, or you’re back to needing custom registry information. In that case my recommendation would be to both rename and change uuid of the forked package and make your own registry for this package.

(*) That statement might be somewhat inaccurate. If you try to add or develop a custom version which is outside of the compat bounds, Pkg will complain. You could specify compat bounds which only are satisfied by your custom version but it wouldn’t help Pkg finding that version unless you add custom registry information.

2 Likes
5

My challenge specifically is that I have the current dependency chain

github.com/bradcarman/OrdinaryDiffEq.jl --> DifferentialEquations.jl --> PkgA.jl --> MainPackage.jl

I have modified a method in OrdinaryDiffEq.jl that is very specific to my needs (maybe in the future I can get a feature request approved, but for now I’m in development mode). I have a team that is using MainPackage.jl which is pulled from a registry, therefore I don’t have any control in this case over the Manifest.toml file. So my choices I can think of are:

  1. Somehow specify the compatibility to the specific package location (github.com/bradcarman/OrdinaryDiffEq.jl). So far I don’t think this is possible.

  2. I can pirate the method definition that I need to modify, so in PkgA.jl I define my own…

nlsolve!(nlsolver::AbstractNLSolver, integrator, cache=nothing, repeat_step=false)

This will give WARNING messages that I’ve overwritten the definition to nlsolve! when PkgA.jl is loaded, but for now technically it works, and the normal registered OrdinaryDiffEq.jl can be loaded and therefore there is no need for a compatibility specification in Project.toml.

So my question is, any better way to do this?

6

A key observation is that only the Manifest file of your activated environment matters and that only one version of a given package can be loaded at the same time. I’m not entirely sure I understand your scenario but if you have an environment with a direct dependency on package A, with the indirect dependency chain A → B → C → D and you want C to bring in a custom version of D, you can simply add the custom version of D as a direct dependency to your environment, even if you don’t use it directly, and then that’s the version C will use.

7

Yes this is true, this might be a workable solution as well. So for my team using MainPackage.jl that is pulled from our private registry, I suppose I could also distribute a Manifest.toml file that they manually place in their working environment. Should work, but maybe a bit messy and hard to maintain over time.

8

If you don’t normally use Manifest files in your workflow it’s probably easiest to go with monkey patching for a temporary solution, if you can do that in a way which is sufficiently clean for your needs. (I’ve been doing that on and off and sometimes precompilation has been an issue.)

For our data processing pipeline we have a repository with a directory containing Project.toml, Manifest.toml and a number of scripts, all starting with

using Pkg
if !isinteractive()
    Pkg.activate(@__DIR__)
    Pkg.instantiate()
end

This works great for reproducibility and would easily allow defining a custom version of some dependency in the manifest.