Hi,
We have upgraded our projects/packages to Julia 1.0. The new Pkg is a well thought-out package manager, and we are currently trying to figure how to best use it in our development workflow. In the following, I describe our scenario first, then explain what workflow we currently have in mind, followed by some questions.
Scenario: We develop an Application Bench, and two unregistered packages Foo and Bar. All three repositories live on github.
- Foo has a dependency on Bar.
- Both packages Foo and Bar also depend on a package registered at General, say JuMP. However, they require a version of JuMP, say 0.19-alpha, that has not been registered in General yet.
- Bench depends on both Foo and Bar. The core function of Bench is to run scientific experiments, which in our case are algorithm benchmarks. For each of the experiments, we would like to specify the exact version, i.e., the commit hash, of Bench, Foo and Bar.
Our current workflow is to run Bench, look at the results, update code in Foo and/or Bar, and re-run Bench. The tricky part now is that Bench parallelizes work over several machines. That is, the package versions of Foo and Bar have to be the same on all of the machines. In the past, we have managed the package version with custom scripts (basically ssh + clone the required version). What we would like to do, however, is to define all of the dependencies in Bench, install/update Bench on each machine, and let the package manager install/update the correct versions of Foo, Bar and JuMP.
This is where we are stuck with several issues/questions, and it took us a while to wrap our heads around this. These are some of our trial/error findings:
Dependency on an unregistered version of a registered package
In our example Foo and Bar require a unregistered version of JuMP (0.19-alpha). We can specify this dependency in the Manifest.toml:
[[JuMP]]
deps = ["Calculus", "Compat", "DataStructures", "ForwardDiff", "MathOptInterface", "NaNMath"]
git-tree-sha1 = "5d83aac41618d428bb97c0f3beecfcae3c49989a"
repo-rev = "a1d333eec8304f15eefc6f9a750ddd975ae319d1"
repo-url = "https://github.com/JuliaOpt/JuMP.jl.git"
uuid = "4076af6c-e467-56ae-b986-b466b2749572"
version = "0.18.4+"
However when installing for example Foo via add https://github.com/user/Bar.jl, we run into the following issue:
ERROR: Unsatisfiable requirements detected for package MLKernels [48eadcf2]:
MLKernels [48eadcf2] log:
├─MLKernels [48eadcf2] has no known versions!
└─restricted to versions * by Foo [83536b18] — no versions left
└─Foo [83536b18] log:
├─possible versions are: 0.2.0 or uninstalled
└─Foo [83536b18] is fixed to version 0.2.0
Current Fix: First manually add the specific JuMP version and then add Bar via the github url. However, this requires us to manage the dependencies manually.
Dependency of an unregistered package on another unregistered package
As both Foo and Bar are unregistered, we currently do not understand how to correctly define a requirement of Foo on a specific hash of Bar. More specifically, we can define the dependency on Bar in the Manifest.toml of Foo that defines
[[Bar]]
deps = ["JuMP"]
git-tree-sha1 = "some-git-tree-sha1"
repo-rev = "master"
repo-url = "https://github.com/some-user/Bar.jl.git"
uuid = "some-uuid"
version = "0.1.0"
However, if we start with an empty Environment and run add https://github.com/user/Foo.jl, the package manager cannot find the package and returns a similar error:
ERROR: Unsatisfiable requirements detected for package Bar ... Bar has no known versions.
Current Fix: Manually add Bar via github url before adding Foo.
So after some issue browsing on Pkg.jl, our current understanding is that Pkg cannot resolve non-registered versions and non-registered packages. The reason is that Pkg would have to resolve the package versions on the local machine, i.e., download the package first, then look at the dependencies, which might be unregistered versions as well. This would require to download a lot of repositories.
Our current solution: This leaves us with the question is how to solve our multi-machine dependency management. We have the following option in mind:
We specify a Manifest.toml in Bench, that states the exact location and revision of Foo, Bar and JuMP. On each of our nodes, we would clone “Bench”, and then run Pkg.instantiate() inside the application directory. To run Bench, we have one driver node that executes something like
@everywhere using Pkg
@everywhere Pkg.activate("Bench")
@everywhere using Foo
Or alternatively pass the project directory during the distributed setup
addprocs(2, exeflags=`--project=Bench`)
This would solve the dependency issues for our scenario. In general, however, if someone wants to use Foo in their project, one has to figure out which version of Bar and JuMP have to be installed, and the location where to find them. This is certainly possible by looking at the Manifest.toml in Foo, but one would have to manage these dependencies manually. This seems to make developing packages that depend on unregistered versions difficult.
This leaves us with the following questions:
- Is our solution described the suggested way to do this? Or is there a better option that we do not see?
- What is the current way of managing dependencies on unregistered packages/versions? A private registry? What is the future of managing such dependencies?
Some of the Pkg issues this post seems to be related to: #698 #492 #823