I wrote this simple tool for generating a Project.toml to make your Julia package compliant with the new Pkg.
Just load up this module (put the file in your LOAD_PATH and do using GenProject, head over to the directory of the project you want to generate files for (i.e. make it your pwd) and do genproject(). This will initialize and Project.toml and Manifest.toml from the packages listed in your REQUIRE and download the dependencies into the new Pkg directories if they aren’t there already.
By default this will fetch the latest tagged version of all the packages in your REQUIRE. I thought this was a good default behavior because we are in an upgrading frenzy, and in most cases you’ll need to do this to get things up and running on 0.7 anyway. If however you want the minimum versions listed in REQUIRE you can do genproject(use_latest=false), but warning! it’s possible that the packages without version tags in REQUIRE will conflict with requirements of older packages that you provided tags for. Undoubtedly there is a way of making sure this always gets resolved using the new Pkg, but I haven’t done that (thus the qualification “quick and dirty”). (It’s highly likely you’ll just need latest versions anway, so I’m not sure this will ever matter).
Another thing missing from this is that it does not download any of the git metadata that ideally would go in the Project.toml (e.g. project description, authors), but you shouldn’t need that to get your package working.
If you find something else missing that you really need, feel free to improve on this and post your own!
One question, for packages already available in METADATA, what should I do during the transitions between Pkg2 to Pkg3? I read somewhere that probably there will be PR to add the correct files to the METADATA packages, is that correct?
METADATA is ultimately going to be deprecated in favor of Registries. The only registry right now (Uncurated) is being automatically synced with METADATA. So, if your package is already in METADATA, you don’t need to do anything. I don’t know what the plans are for how new packages are to be added to Uncurated in the future, presumably much the same way as they are added to METADATA now.
Thanks, this seems useful. However, I’m not able to install it and I’m not sure why. I cloned the gist to ~/Code/GenProject/GenProject.jl, then do
(v0.7) pkg> add /Users/cbinz/Code/GenProject
Cloning git-repo `/Users/cbinz/Code/GenProject`
Updating git-repo `/Users/cbinz/Code/GenProject`
┌ Warning: packages will need to have a [Julia]Project.toml file in the future
└ @ Pkg.Types Types.jl:592
[ Info: Assigning UUID a21259ae-7a31-11e8-34d7-b9d871f2d3f9 to GenProject
...
but then
julia> using GenProject
ERROR: ArgumentError: Package GenProject [a21259ae-7a31-11e8-34d7-b9d871f2d3f9] is required but does not seem to be installed:
- Run `Pkg.instantiate()` to install all recorded dependencies.
Stacktrace:
[1] _require(::Base.PkgId) at ./loading.jl:951
[2] require(::Base.PkgId) at ./loading.jl:879
[3] require(::Module, ::Symbol) at ./loading.jl:874
I know this is a more general problem with how I’m using Pkg, but do you see what I’m doing wrong? I find Pkg3 really difficult to wrap my head around, so far.
I’m honestly not sure what behavior I expected from the package manager for adding just this file. Try just adding the directory the file is in to the LOAD_PATH and do using GenProject, that’s how I did it.
It may be a moot point now anyway I know that @StefanKarpinski and @kristoffer.carlsson are working on a “real” version of this for part of Pkg itself, I’m not sure how far along they are.
I believe that the uuid’s for existing packages, which are already assigned and in the registry, are not going to change. Look in the directory ~/.julia/registries/Uncurated which has subdirectories based on the first letter of the package name. Thus
Sorry I was not clear: I wanted to generate an UUID (and the rest of the header) for an existing, but unregistered package. I understand that the UUID in particular is somewhat random, but AFAICT there are various options and I was not sure which one to choose to generate with the stdlib UUIDs.
They are 128 bit so my understanding is that they are not checked for collisions. I think you can just use the UUIDs module to generate one, however at the moment I don’t remember what to do with it once you have it.
Apologies if I’m misunderstanding you, but wouldn’t ] generate MyPackageName in some temporary directory get you a header that you can copy into your existing package (of the same name)?
UUIDs are a bit of a black box to me, apparently there is uuid1 and uuid4, and some information is incorporated into the hash, so I am not sure if I can use something completely arbitrary.
Yes, that’s what I ended up doing, but it would be nice to have a more direct way. Something that can take an existing package with a v0.6-style REQUIRE, and just generate the full Project.toml. @ExpandingMan’s nice tool gets you almost here, except for the header.
Is there a “proper” version of this yet? (i.e. that correctly generates name UUID and other metadata) It seems to me that having this would be an important part of the upgrade work flow. The reason for this is that right now one often has to work with the latest master, and the only way I can see for working on the latest master for all the indirect dependencies would be to activate it and do develop Dependency everywhere. Is there some simpler way of doing this that I’m missing?
It might be nice if there were something like developall PackageName which sets everything to use master, this would simplify testing and cases like this where we are upgrading almost everything at once.
pkg> dev https://github.com/JuliaLang/PkgDev.jl#kc/07
julia> import PkgDev
julia> PkgDev.generate("path/to/my/pacakge", "MIT"; force=true)
and it will try to update CI scripts, make a Project file etc.
But please note the project format is likely slightly changing soon (but not dramatically). When I am done with what is needed for Pkg itself for 0.7 I will work on PkgDev for 0.7.
Maybe I’m just confused. I thought that without the Project.toml everything was getting the most recent tagged version. So for example, I thought that if I did dev DataFrames then using DataFrames it would use the most recent tagged versions of all the DataFrames dependencies and that the only way to get it to use the latest masters would be to have a Project.toml so that I could activate DataFrames and then do up to the dependencies. Am I remembering this wrong, I haven’t actually done this in the last week or so but I could have sworn that this is what happened?
That seems a bit confused yes. DataFrames can set compatibility bounds on the dependencies but it doesn’t decide what version of the dependencies it will have, the current project does. So if you have DataFrames installed and you do dev SortingAlgorithms then DataFrames will load the devved version of SortingAlgorithms. If you do pkg> status -m you can see all the versions that will be loaded.
Ok, I’ve figured out why I was so confused. When something compiles I often see it saying that it’s compiling a version with a hashed name like PackageName/AR9o.ji. Within packages/PackageName there are similarly hashed names, but with dev you don’t see them. So when I was seeing those names I thought I was seeing packages from the package directory rather than the dev directory being compiled (I guess I was expecting them to be called PackageName/dev.ji or something) as I didn’t know what the hashed version designation for the latest master was.