LightGraph tutorial

General Usage
1

I am trying to replicate this LightGraph tutorial. Right at the start it has this snippet of code:

pkgs = readdir(Pkg.dir("METADATA"))
filterfunc = p -> isdir(joinpath(Pkg.dir("METADATA"), p)) && p ∉ [".git",".test"]
pkgs = filter(filterfunc, pkgs)

Julia tells me that Pkg.dir is deprecated and it returns nothing. What is the first line of this code supposed to do and how can I replicate this on Julia 1.2?

2

METADATA was the old way that the old package system kept track of registered packages. IIRC, it contained a folder for each registered package, and I think those first lines of code are just getting all of the package names by reading the contents of the folder.

There are a bunch of old Pkg features that tutorial uses that are no longer relevant, but Pkg3 (the new package manager) can probably be leveraged to do the same work. Alas, I don’t know enough to help much, but what you want to do is

  1. Get a list of all packages in the General registry (that’s analogous to the old METADATA)
  2. Parse their Project.toml files to get their dependences

Good luck!

1 Like
3

It’s always worth looking in the Issues for a project to see if there are any known problems and work rounds.

In this case, there are known problems but no work rounds as yet. :confused:

4

This is unlikely to be the proper way to do it but it should give you the corresponding list of packages:

using Pkg.TOML: parsefile
registry = parsefile(joinpath(first(DEPOT_PATH), "registries", "General", "Registry.toml"))
pkgs = [package["name"] for package in values(registry["packages"])]

As already mentioned this is only the first step and there are more things that have changed with the package manager. However, all necessary information to build the package DAG is available in the same directory and can be obtained using parsefile on the appropriate TOML files in there. An earlier discussion may also be helpful to get further: Replacement for Pkg.dependents() in 1.0?

5

Thanks, that helped a lot. I parsed the Deps.toml files of the packages and I have constructed the DAG. However, I have included dependencies that have been dropped because I didn’t find an easy way to exclude them.

By the way, through the graph I found that the 10 most “important” packages (not included in julia by default) are in order: Tokenize, CSTParser, JuliaFormatter, DocumentFormat, AutomaticDocstrings, SourceWalk, MacroTools, SimpleTraits, ZygoteRules and Unrolled. :slight_smile:

1 Like
6

That’s great! You should totally make a PR to update the notebook!

7

Unfortunately I don’t know how to deal with notebooks and I don’t have time to learn now. I’m dumping my code here and anyone who feels like updating the example should feel free to use it. :slight_smile:

using Pkg.TOML: parsefile
using LightGraphs

registry = parsefile(joinpath(first(DEPOT_PATH), "registries", "General", "Registry.toml"))
pkgs = [package["name"] for package in values(registry["packages"])]

pkgdict = Dict{String,Int}()
pkgdictinv = Dict{Int,String}()
for (i,pkg) in enumerate(pkgs)
  push!(pkgdict, pkg => i)
  push!(pkgdictinv, i => pkg )
end

DAG = DiGraph(length(pkgs))

pth = joinpath(first(DEPOT_PATH), "registries", "General")
lttrfolders = readdir(pth)
pathfilterfunc = p -> isdir(joinpath(pth, p)) && p[1] != '.'
lttrfolders = filter(pathfilterfunc, lttrfolders)
for lttr in lttrfolders
  ps = readdir(joinpath(pth, lttr))
  for p in ps
    try
      lns = readlines(joinpath(joinpath(pth, lttr), p, "Deps.toml"))
    catch e
      println(e)
      continue
    end
    linefilterfunc = p -> p != "" && p[1] != '['
    lns = filter(linefilterfunc, lns)
    deps = [split(l," ")[1] for l in lns]
    for dep in deps
      try
        if pkgdict[p] == pkgdict[dep]
          println(p)
        else
          try
            add_edge!(DAG, pkgdict[dep], pkgdict[p])
          catch e
            println(e)
            return
          end
        end
      catch e
        # println(e)
      end
    end
  end
end