Convert REQUIRE to Project.toml and Manifest.toml

armstrtw · November 20, 2018, 2:25pm

Is there a standard process or function that allows one to upgrade old packages using the REQUIRE file to the new package format using Project.toml and Manifest.toml ?

Apologies if I missed this documentation online.

mohamed82008 · November 21, 2018, 12:12am

This is what I do:

Steal the format from an existing Project.toml, e.g. https://github.com/JuliaLang/Pkg.jl/blob/master/Project.toml, and create your own Project.toml in your repository.
You probably only need “name” and “uuid” in the top section, so you can delete the rest or change it for your package if you want.
Replace the package name there with your package name.
Get a uuid for your package using using Pkg; Pkg.METADATA_compatible_uuid(pkg_name) where pkg_name is the name of your package as a String without .jl (watch your spelling!).
Replace the uuid in the Project.toml and delete the list of deps there, keeping the header. Save the file!
cd into the repository of the package, most likely in your .julia/dev/pkg_name folder.
In a Julia session, run ]activate .
Open the REQUIRE file in another window.
Run ]add dep_name for all your dependencies in the REQUIRE file. This will automatically update the Project.toml and Manifest.toml, creating the latter if it doesn’t exit.
You may want to add an extra [compat] section as shown in Pkg · The Julia Language to support certain versions of packages or Julia.
Add your test dependencies as well and then copy the test deps’ lines from the Project.toml file’s [deps] section to the [extras] section, and list the package names in the [targets] section as shown in https://github.com/KristofferC/JuAFEM.jl/blob/master/Project.toml.

And now you should be having Project.toml and Manifest.toml files in your repo. Apparently, you still need to keep your REQUIRE files there for registered packages. Anyways after doing the above, you can close that Julia session and open a new one in the default v1.0 environment, and pretend like none of this had to happen. Hopefully, some expert can correct me if I got something off.

tkf · November 21, 2018, 8:41am

Here is my quick-and-dirty Julia function to automate a similar workflow:

"""
    generate_project_toml([name::String])

Generate Project.toml file for the existing current project at `\$PWD`.
It activates the generated Project.toml and then adds packages based on
REQUIRE file.
"""
function generate_project_toml(name::AbstractString = guess_project_name())
    if isfile("Project.toml")
        error("Project.toml exists.")
    end

    if !isfile("REQUIRE")
        error("REQUIRE file not found.")
    end
    deps = read_require("REQUIRE")

    pkg = Base.identify_package(name)
    project_toml = """
    name = $(repr(name))
    uuid = "$(pkg.uuid)"
    """
    write("Project.toml", project_toml)

    @info "Activating \$PWD (to run `Pkg.add`)"
    Pkg.activate(pwd())
    Pkg.add(deps)
end

"""
    guess_project_name() :: String

Guess project name using Git.  It works even in worktree repository.
"""
function guess_project_name()
    path = abspath(rstrip(read(`git rev-parse --git-dir`, String)))
    prev = ""
    while path != prev
        if basename(path) == ".git"
            return basename(dirname(path))
        end
        prev, path = path, dirname(path)
    end
    error("Project not found")
end

function read_require(path)
    deps = String[]
    for line in readlines(path)
        name, = split(line)
        if name == "julia"
            continue
        end
        push!(deps, name)
    end
    return deps
end

dehann · February 11, 2019, 4:04pm

I used tkf’s functions (thanks!), but had to ask for more help to resolve “core” package dependency issues including LinearAlgebra, Statistics, Random, Test, etc. Maybe it has already been mentioned somewhere but I missed it. Turns out the fix is easy and just noting it here for documentation’s sake:

Start Julia with working directory in the project repo, for example cd ~/.julia/dev/MyPkg. After running generate_project_toml("MyPkg") here, open the package manager ]; then type activate .. This should show (MyPkg) pkg>, and now add the missing packages such as add LinearAlgebra Statistics .... This will append to the deps section in Project.toml and resolve issues you might discover in Travis.

Aside: make sure the MyPkg/.travis.yml file is also updated to the newer script: format:

github.com

JuliaLang/Pkg.jl/blob/e8414f1a2409b31a0b10859894ec0cbe11f2a85d/.travis.yml#L32


      
              - /^v[0-9]+\.[0-9]+\.[0-9]+$/ # version tags
          
          
stages:
            - name: test
              # test only on bors trying and bors staging
              if: branch IN (trying, staging)
            - name: docs
              # build docs on bors trying, bors staging, master and for tags
              if: branch IN (trying, staging) OR (branch = master AND type = push) OR tag IS present
          
          
script:
            - if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
            - julia --project --check-bounds=yes -e 'using UUIDs; write("Project.toml", replace(read("Project.toml", String), r"uuid = .*?\n" =>"uuid = \"$(uuid4())\"\n"));
                                                     import Pkg; Pkg.build(); Pkg.test(; coverage=true)'
          
          
jobs:
            include:
              - stage: docs
                julia: nightly
                os: linux
                script:

Wikunia · April 18, 2019, 2:24pm

Can you specify on point 10? I can’t find anything about [compat] in that link

fredrikekre · April 18, 2019, 2:25pm

https://julialang.github.io/Pkg.jl/v1/compatibility/

arumugamav · July 30, 2019, 4:34am

Please guide me in constructing site Project.toml from packages, Project.toml!

kevbonham · July 30, 2019, 11:22am

This is how I did it:

arumugamav · August 1, 2019, 4:36am

Thanks!
I already have Project.toml in my package folders.
(ie. /opt/julia/julia-1.1.0/share/julia/stdlib/v1.1/<PACKAGE_NAME>/Project.toml files are there)
But I don’t have user level/host level Project.toml and Manifest.toml (ie.“/opt/julia/julia-1.1.0/share/julia/stdlib/v1.1/Project.toml” or /home/ava/.julia/)

May I request your guidance to construct consolidated single Project.toml and Manifest.toml at host level!

kevbonham · August 1, 2019, 10:50am

Not sure I understand exactly what you’re looking for, but if you mean getting them for your global environment, it’s as simple as ] add OhMyREPL (or whatever packages you want). Alternatively,

julia> using Pkg

julia> Pkg.add("OhMyREPL")

See here and here.

arumugamav · August 2, 2019, 4:59am

Thanks!

It is observed that output of
pkg>status
…
[34da2185] Compat v1.4.0
[31c24e10] Distributions v0.16.4
[2fe49d83] Expectations v1.0.2
[28b8d3ca] GR v0.36.0
[7073ff75] IJulia v1.14.1
[43edad99] InstantiateFromURL v0.1.0
[a98d9a8b] Interpolations v0.11.0
[d96e819e] Parameters v0.10.2
[91a5bcdd] Plots v0.21.0
[fcd29c91] QuantEcon v0.15.0
[295af30f] Revise v0.7.13

What is the 8 character alpha-numeric string preceding package name and how it is arrived?

carstenbauer · August 2, 2019, 6:01am

It’s the first part of the UUID of a package, see the Pkg.jl documentation for more information. Parameters.jl, for example, has the UUID d96e819e-fc66-5662-9728-84c9c7592b0a (Project.toml).

You can find a package’s UUID by looking it up in the general registry. To generate a UUID, you can use

julia> using Pkg

julia> Pkg.METADATA_compatible_uuid("MyPackage.jl")
UUID("7bec01f6-6e58-5d42-b8c7-b05f3435ceef")

kevbonham · August 2, 2019, 11:18am

Alternatively, if you don’t need your package to work on julia prior to v1.0:

julia> using UUIDs

julia> uuid4()

astadmistry · September 3, 2020, 2:22am

I have updated the script to work with Julia 1.5
When I ran the script, I was getting:

ERROR: LoadError: type Nothing has no field uuid

import Pkg
import Base: SHA1
using SHA, UUIDs

function uuid5(namespace::UUID, key::String)
    data = [reinterpret(UInt8, [namespace.value]); codeunits(key)]
    u = reinterpret(UInt128, sha1(data)[1:16])[1]
    u &= 0xffffffffffff0fff3fffffffffffffff
    u |= 0x00000000000050008000000000000000
    return UUID(u)
end

uuid5(namespace::UUID, key::AbstractString) = uuid5(namespace, String(key))

const uuid_dns = UUID(0x6ba7b810_9dad_11d1_80b4_00c04fd430c8)
const uuid_julia_project = uuid5(uuid_dns, "julialang.org")
const uuid_package = uuid5(uuid_julia_project, "package")



"""
    generate_project_toml([name::String])

Generate Project.toml file for the existing current project at `\$PWD`.
It activates the generated Project.toml and then adds packages based on
REQUIRE file.
"""
function generate_project_toml(name::AbstractString = guess_project_name())
    pkg_UUID = uuid5(uuid_package, name)
    if isfile("Project.toml")
        error("Project.toml exists.")
    end

    if !isfile("REQUIRE")
        error("REQUIRE file not found.")
    end
    deps = read_require("REQUIRE")

    pkg = Base.identify_package(name)
    project_toml = """
    name = $(repr(name))
    uuid = "$(pkg_UUID)"
    """
    write("Project.toml", project_toml)

    @info "Activating \$PWD (to run `Pkg.add`)"
    Pkg.activate(pwd())
    Pkg.add(deps)
end

"""
    guess_project_name() :: String

Guess project name using Git.  It works even in worktree repository.
"""
function guess_project_name()
    path = abspath(rstrip(read(`git rev-parse --git-dir`, String)))
    prev = ""
    while path != prev
        if basename(path) == ".git"
            return basename(dirname(path))[1:end-3]
        end
        prev, path = path, dirname(path)
    end
    error("Project not found")
end

function read_require(path)
    deps = String[]
    for line in readlines(path)
        name, = split(line)
        if name == "julia"
            continue
        end
        push!(deps, name)
    end
    return deps
end

Github gist of above script

REF:
https://github.com/JuliaLang/Pkg.jl/commit/34ec22f5405ffc7f8f25a4b4f7178e887f718051
https://github.com/JuliaLang/Pkg.jl/blob/34ec22f5405ffc7f8f25a4b4f7178e887f718051/src/Types.jl
https://github.com/JuliaLang/Pkg.jl/search?p=1&q=gen_project.jl&type=Issues

Tamas_Papp · September 3, 2020, 9:00am

Scripts like the above rely on internals and will inevitably break over time.

Given that it is 2 years past the release of Julia 1.0, generating the Project.toml with a series of pkg> adds for the odd package that has not been ported yet may be the best approach.

astadmistry · September 16, 2020, 2:20pm

The above script is…

as seen on line 48

The vast majority of the script is to produce a UUID which is backwards compatible for versions of julia prior to v1.0

Tamas_Papp · September 18, 2020, 8:29am

Sorry, I don’t understand what you want to convey here. My point was that even if the script no longer works (for whatever reason), generating a project file manually is a reasonable alternative, especially since it is only rarely needed these days as the majority of maintained packages did it already.

Also, keep in mind that for the General registry, you need [compat] bounds in any case, which need to be given some thought.

The bottom line is that reviving packages that missed the transition to Julia 1.0 may be tricky to automate at this point, but at the same time it is still fairly easy to do it manually.