The new Pkg3 package manager is now in Base and fairly feature complete, although still somewhat sparsely documented. I’m working on the latter and @kristoffer.carlsson and others have been doing truly epic work on adding and improving features now that we have it hooked into Base. So here’s the status and where we’re going:
The main issue preventing using Pkg3 right now is that Pkg3 is 0.7-only and there aren’t very many packages that are 0.7 compatible yet. As that changes, we encourage people to try it out and report any issues, annoyances, or ideas that they have from using it.
Once 1.0-alpha is out, most packages add support for it and we’re confident that we’ve got most of the kinks in the release and in Pkg3 worked out, people should generally start using Pkg3 and stop using Pkg (we should probably rename it to Pkg2 so that we can rename Pkg3 to Pkg in 1.0 final).
Currently, the old METADATA repo is still the definitive source of all package version, dependency and compatibility information. The data in METADATA is automatically synced to the new Uncurated registry hourly so that Pkg3 knows about all the packages and versions that are in METADATA.
However, there is not yet any way to express the compatibility part of REQUIRE files in Pkg3. I need to work that out before we switch over from METADATA being definitive to the Uncurated registry being definitive. There’s been a lot of design debate and discussion of how to do this, from something basic resembling other package managers to more radical ideas. More to come on this.
We will, at some point need to make a bunch of PRs automatically to all registered Julia packages, adding a Project.toml file giving each package a name, UUID, dependency map, etc. We could probably do that at any point, but there’s no real urgency and if we change anything we’ll just have to make new PRs to change the files we already made, which may have been manually modified since then, so there’s no point in doing this until we absolutely have to.
Once we have a way to express everything we need to in the Uncurated registry and have tooling for that at parity with METADATA, we can switch to having Uncurated be definitive, and maybe propagate backwards from Uncurated to METADATA. It’s unclear at this point what the timeline will be and it’s possible that at the point when this happens we’ll just stop updating METADATA altogether. Hard to say.
Hopefully that clarifies where Pkg3 is and where it’s going. Stay tuned for improved documentation which should significantly help people understand how Pkg3 works and how to use it.
During the transition, will we be able to clean repos and/or delete old tags? This would finally fix the DifferentialEquations.jl and Plots.jl repo size issue.
The new design inherently makes that not an issue since package versions are associated with trees not commits. Any way you get the right tree is fine. By default versions are downloaded as GitHub tarballs.
This is surely good for users, that won’t have to keep bloated git repository on their system. Regarding developers, would it be still possible for them to use git repositories as packages?
Moreover, you can rewrite or replace a repo with a new one with entirely different history as long as the git trees needed for specific versions exist. So we could now rewrite the history of repos with historical large objects to discard those objects.
Could someone say what kind of behavior they are expecting for the keywords search?
Either, one could match for exact equality for the keywords, or one can match for a substring, don’t know which people expect or prefer.
Would also like to know how the package directories are going to be stored, since multiple packages with same name are possible. Will we continue to have ~/.julia/v1.0 or some sub directory of that? I understand how the registries / depots will be organized, but I’d like to understand better how the actual packages will be saved
If you are only using them they will be stored somewhere “hidden” which right now is .julia/packages/.... These are read only packages though. If you are developing a package, you can have it anywhere you want and you will use e.g. develop ~/myjuliastuff/MyCoolPkg.
This seems like a good example of why the Pkg system needs to have per-platform metadata.
e.g. If MbedTLS 0.5.7 had “validated platform” tags for “-darwin”, “-mingw32”, “-linux” (but not for “-freebsd”) then Pkg.add would know to use an older version on freebsd.
It seems that as the number of supported platforms grows it will be increasingly common for a package maintainer to be unable to test every platform (embedded systems, systems not supported by Travis etc). In this case it makes sense to push out a release with “validated platform” tags for the platforms that are covered by CI, and let the community add tags for other platforms one by one as they are tested off line.
Is there a plan for how this will eventual work with Pkg3?
Great news. Can you please clarify the status of Pkg3 in stdlib? Is it going to be pinned to a particular version at the time of release, or can one upgrade it to the latest Pkg3 to keep up with improvements? My understanding is that Pkg3 itself should allow multiple versions for any package, incl itself.
No, I dont think it will be pinned to a version, it can be updated.
However, with the validated platform metadata, what if a package has not been tested on a platform, but if actually turns out to work perfectly fine on that platform, you don’t want to stop that package from being installable just because nobody tested it yet. So there must be a way to force the install, even if the tag is missing for that particular platform.
The plan is to have stdlibs be updatable like normal packages. It is not completely clear yet exactly how because there are some issues that need to be figured out first. Load times have to be acceptable for example so perhaps some sysimg recompilation is needed etc.
I’m not sure what this means. What do you mean with package tested on platform?
My post was in response to the issue @samoconnor mentioned with having meta data for platform specific testing. The quote he posted mentions having some features for preventing a new package to be updated or installed of it has not been tested on some platform. However, this seems problematic, because a package may not be tested for a platform, but may be fully functional. So if the package manager prevents installation of a package because it was not tested by the developer on a platform, that does seem problematic. You’d have to also be able to force the install, even if it has not been tested, because it could be that the package works fine on that platform, the developer merely was unable to test
In 1.0 all stdlibs, including Pkg3, are fixed. In some 1.x version, however, we will change that and make it possible to use different stdlib package versions than the Julia version. Since dependencies on stdlibs must be explicitly recorded in your projects already in 1.0, and it’s possible to record the version of a stdlib package that your project depends on in its manifest file, once we support using different stdlib versions, that will allow you to upgrade Julia itself to version 1.x and keep using older versions of stdlib packages – which will keep working because that’s the semver contract for minor updates of Julia. (This may require some trickery in newer stdlib versions since some of them are quite a bit more deeply entangled with Base than typical packages, but fortunately, this kind of thing is fairly doable in a dynamic language.)
This plan allows stdlibs to make breaking API changes while old Julia programs can keep working with newer 1.x versions of Julia by using older versions of stdlibs. This way we follow semver without preventing stdlibs from evolving during the 1.x Julia release cycle. This means that newer versions of stdlibs may have higher version numbers than Julia itself. For example, Julia 1.5 could ship with Dates 2.0 which makes breaking API changes to the Dates stdlib package. This isn’t a semver violation because projects that were written to use Dates 1.4 can continue to do so even while upgrading Julia itself to 1.5.
Sure, seems like a good idea but probably best as an external integration. There’s no need for Pkg3 itself to know or care about this. The [misc] section in Project.toml is guaranteed not to be significant to Pkg3 so there could be a [misc.zenodo] subsection that allows packages to provide whatever metadata is necessary for citation with Zenodo.