A quick and dirty tool for generating `Project.toml`

announcement
pkg

#1

I wrote this simple tool for generating a Project.toml to make your Julia package compliant with the new Pkg.

Just load up this module (put the file in your LOAD_PATH and do using GenProject, head over to the directory of the project you want to generate files for (i.e. make it your pwd) and do genproject(). This will initialize and Project.toml and Manifest.toml from the packages listed in your REQUIRE and download the dependencies into the new Pkg directories if they aren’t there already.

By default this will fetch the latest tagged version of all the packages in your REQUIRE. I thought this was a good default behavior because we are in an upgrading frenzy, and in most cases you’ll need to do this to get things up and running on 0.7 anyway. If however you want the minimum versions listed in REQUIRE you can do genproject(use_latest=false), but warning! it’s possible that the packages without version tags in REQUIRE will conflict with requirements of older packages that you provided tags for. Undoubtedly there is a way of making sure this always gets resolved using the new Pkg, but I haven’t done that (thus the qualification “quick and dirty”). (It’s highly likely you’ll just need latest versions anway, so I’m not sure this will ever matter).

Another thing missing from this is that it does not download any of the git metadata that ideally would go in the Project.toml (e.g. project description, authors), but you shouldn’t need that to get your package working.

If you find something else missing that you really need, feel free to improve on this and post your own!


Advice on creating packages in v0.7
#2

One question, for packages already available in METADATA, what should I do during the transitions between Pkg2 to Pkg3? I read somewhere that probably there will be PR to add the correct files to the METADATA packages, is that correct?


#3

METADATA is ultimately going to be deprecated in favor of Registries. The only registry right now (Uncurated) is being automatically synced with METADATA. So, if your package is already in METADATA, you don’t need to do anything. I don’t know what the plans are for how new packages are to be added to Uncurated in the future, presumably much the same way as they are added to METADATA now.


#4

Thanks, this seems useful. However, I’m not able to install it and I’m not sure why. I cloned the gist to ~/Code/GenProject/GenProject.jl, then do

(v0.7) pkg> add /Users/cbinz/Code/GenProject
   Cloning git-repo `/Users/cbinz/Code/GenProject`
  Updating git-repo `/Users/cbinz/Code/GenProject`
┌ Warning: packages will need to have a [Julia]Project.toml file in the future
└ @ Pkg.Types Types.jl:592
[ Info: Assigning UUID a21259ae-7a31-11e8-34d7-b9d871f2d3f9 to GenProject
...

but then

julia> using GenProject
ERROR: ArgumentError: Package GenProject [a21259ae-7a31-11e8-34d7-b9d871f2d3f9] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
 [1] _require(::Base.PkgId) at ./loading.jl:951
 [2] require(::Base.PkgId) at ./loading.jl:879
 [3] require(::Module, ::Symbol) at ./loading.jl:874

I know this is a more general problem with how I’m using Pkg, but do you see what I’m doing wrong? I find Pkg3 really difficult to wrap my head around, so far.


#5

I’m honestly not sure what behavior I expected from the package manager for adding just this file. Try just adding the directory the file is in to the LOAD_PATH and do using GenProject, that’s how I did it.

It may be a moot point now anyway I know that @StefanKarpinski and @kristoffer.carlsson are working on a “real” version of this for part of Pkg itself, I’m not sure how far along they are.


#6

This works for me:

(v0.7) pkg> dev GenProject
  Updating `~/.julia/environments/v0.7/Project.toml`
  [927275f4] + GenProject v0.0.0 [`~/Documents/GenProject`]
  Updating `~/.julia/environments/v0.7/Manifest.toml`
  [927275f4] + GenProject v0.0.0 [`~/Documents/GenProject`]

julia> using GenProject

julia>

#7

This is a nice tool, but is there a way to generate the header part (package name, uuid, etc) for an existing project?


#8

I believe that the uuid’s for existing packages, which are already assigned and in the registry, are not going to change. Look in the directory ~/.julia/registries/Uncurated which has subdirectories based on the first letter of the package name. Thus

$ cat ~/.julia/registries/Uncurated/D/DataFrames/Package.toml 
name = "DataFrames"
uuid = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
repo = "https://github.com/JuliaData/DataFrames.jl.git"

#9

Stefan posted a way (added it to the API even) to get the same uuid that the METADATA -> registry code does.


#10

Sorry I was not clear: I wanted to generate an UUID (and the rest of the header) for an existing, but unregistered package. I understand that the UUID in particular is somewhat random, but AFAICT there are various options and I was not sure which one to choose to generate with the stdlib UUIDs.


#11

They are 128 bit so my understanding is that they are not checked for collisions. I think you can just use the UUIDs module to generate one, however at the moment I don’t remember what to do with it once you have it.


#12

Apologies if I’m misunderstanding you, but wouldn’t ] generate MyPackageName in some temporary directory get you a header that you can copy into your existing package (of the same name)?


#13

UUIDs are a bit of a black box to me, apparently there is uuid1 and uuid4, and some information is incorporated into the hash, so I am not sure if I can use something completely arbitrary.

Yes, that’s what I ended up doing, but it would be nice to have a more direct way. Something that can take an existing package with a v0.6-style REQUIRE, and just generate the full Project.toml. @ExpandingMan’s nice tool gets you almost here, except for the header.


#14

Reading the Pkg.jl source code, I think that Pkg.API.project generates a Project.toml file with just a header.