Script to find which of your dependencies is not working in julia 0.7

Tooling
1

So obviously before you can update your package to be compatible with julia 0.7,
you need to update any of your dependencies to julia 0.7

I have put together a script
which recursively runs Pkg.test on the specified package and then (if it fails) on its dependencies.
And produces a report.


using Pkg

const _tested = Dict{String, Bool}()

function passing_all_tests(pkgname)
	get!(_tested, pkgname) do
		try
			Pkg.add(pkgname)
			Pkg.test(pkgname)
			return true
		catch
			return false
		end
	end
end

function get_deps(pkgname)
	pkg_dir = Pkg.dir(pkgname)
	@show (pkgname, pkg_dir)
	reqfile = joinpath(pkg_dir, "REQUIRE")
	if !isfile(reqfile)
		# anything without a REQUIRE file is not going to be a problem anyway.
		#This mostly occurs if you are testing something in stdlib
		return String[]
	end
	deps = String[]
	for line in eachline(reqfile)
		(strip(line)=="" || line[1]=='#' || line[1]=='@') && continue 
		#^ Skip comments and things like @osx
		
		pkg_name = split(line)[1]
		pkg_name == "julia" && continue
		push!(deps, pkg_name)
	end
	deps
end

function check(pkgname, final_output=stdout, depth=0)
	if !passing_all_tests(pkgname)
		println(final_output, "\t"^depth * pkgname)
		# don't both checking dependencies of things that pass
		
		# Can not use the below with an asyncmap, or it gets weird
		for depname in get_deps(pkgname)
			check(depname, final_output, depth+1)
		end
	end
end


function deptree_report(pkgname)
	final_output = IOBuffer()
	println(final_output)
	println(final_output, "-"^70)
	println(final_output, "Failing Dependency Tree  for " * pkgname)
	println(final_output, "-"^70)
	
	
	check(pkgname, final_output)
	
	println(final_output, "-"^70)
	println(String(take!(final_output)))
end

Copy and paste it into your repo and call it via: deptree_report("MyPackage").
it will dump out a huge pile of text from all the tests it is calling.
Then at the end it will print out a report.
Any package named in that report is failing its tests on your system.
That might be its own fault, or it might be the fault of its dependencies (esp) if they are listed after. Or both.
(Also failing locally doesn’t nesc mean it is broken on 0.7 for everyone; in the examples below SHA is failing on my computer because of a file-permission error due to where it is installed.)

Also running it for many packages you want to test, one after the other is good,
as it caches test results, so you can avoid rerunning tests.
(Conversely test results won’t updated without a restart/clearing the global _tested Dict)

Note this script it will explicitly install all your package’s dependencies, you can probably run it in a throw-away environment if you want.
Also this can take a while, and you might want to turn off depwarn.

Here are some example outputs:

MacroTools

Everything is fine.

julia> deptree_report("MacroTools")
...
----------------------------------------------------------------------
Failing Dependency Tree  for MacroTools
----------------------------------------------------------------------
----------------------------------------------------------------------

Flux

----------------------------------------------------------------------
Failing Dependency Tree  for Flux
----------------------------------------------------------------------
Flux
        DataFlow
        Requires
        Adapt
        GZip
        Colors
                ColorTypes
                        FixedPointNumbers
                FixedPointNumbers
        ZipFile
        AbstractTrees
----------------------------------------------------------------------

TensorFlow

Failing Dependency Tree  for TensorFlow
----------------------------------------------------------------------
TensorFlow
        ProtoBuf
        PyCall
                Conda
                        BinDeps
                                URIParser
                                SHA
        TakingBroadcastSeriously
        Conda
                BinDeps
                        URIParser
                        SHA
        Distributions
                StatsFuns
                        Rmath
                                BinDeps
                                        URIParser
                                        SHA
        StatsFuns
                Rmath
                        BinDeps
                                URIParser
                                SHA
        JLD
                HDF5
                        BinDeps
                                URIParser
                                SHA
                        Blosc
                                BinaryProvider
                                        SHA
                                CMakeWrapper
                                        BinDeps
                                                URIParser
                                                SHA
                FileIO
                LegacyStrings
        FileIO
        MNIST
----------------------------------------------------------------------
22 Likes
2

Very nice! It would be worth integrating something like this into the test functionality of the package manager itself.

6 Likes
3

I love this. Would it be possible to compact the output some when a broken dependency has already been displayed? I.e. in the above example, where BinDeps is displayed 5 times, simply having BinDeps* after the first would be sufficient, and maybe when there is only one broken dependency, possibly fit it on the same line?)
(and maybe for 2 or 3 broken dependencies that are leaves, or can be displayed with *), also on same line?

For example:

TensorFlow
    ProtoBuf
    PyCall => Conda => BinDeps => (URIParser, SHA)
    Conda*
    Distributions => StatsFuns => Rmath => BinDeps*`
    StatsFuncs*
    JLD
        HDFS
              BinDeps*
              Blosc
                     BinaryProvider => SHA*
                     CMakeWrapper => BinDeps*
        FileIO
        LegacyStrings
    FileIO
    MNIST

(I’d be happy to help out trying to make this look nicer, if you’d like, after appropriate bikeshedding over what a good “compact” format should look like.

Does Pkg have a nice way currently of showing the dependency tree like this (not just the failing ones)?
That would also be useful, not just part of the test functionality.

1 Like
4

It’s a fairly simple script, should be easy to tweak for output wanted.

I was thinking about only showing things the first time, but that would hide the info that might be needed to determine what to do to fix on of the later ones.

Perhaps the ultimate solution would be to display the DAG, but not sure if there is a good text format for that

5

Well, you would only need to fix BinDeps once, to fix all of them that are later marked as “BinDeps*” for compactness. The output might also need some way of displaying the UUID (or the first part of it), if there are multiple packages with the same name (but different UUIDs), since Pkg3 allows that.

6

The problem is that the script represents that data as a Tree, while what @ScottPJones wants is a Graph. You could solve his issue by representing the data as a Graph instead of a Tree.

7

Yes, but displaying a graph in plain-text is hard.

I am fairly sure calculating good node placement for a layered visualization of a DAG is not NP-Hard.
But it is nontrivial.
And getting that to look good (or readable) when displaying with characters is just not going to be easy

It might be a fun project for someone who is interested in that.

8

has recursive dependency testing like this been added to Pkg3 yet? the docs don’t seem to mention it

9 
 Warning: Pkg.dir is only kept for legacy CI script reasons
â”” @ Pkg.API API.jl:431
ERROR: type Nothing has no field captures
Stacktrace:
 [1] getproperty at .\sysimg.jl:18 [inlined]
 [2] splitdrive(::String) at .\path.jl:37
 [3] joinpath(::String, ::String) at .\path.jl:217
 [4] entry_point_and_project_file(::String, ::String) at .\loading.jl:429
 [5] implicit_project_deps_get(::String, ::String) at .\loading.jl:583
 [6] project_deps_get at .\loading.jl:320 [inlined]
 [7] identify_package(::String) at .\loading.jl:253
 [8] dir(::String) at .\logging.jl:317
 [9] get_deps(::String) at .\REPL[4]:2
 [10] check(::String, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Int64) at .\REPL[5]:7
 [11] check at .\REPL[5]:2 [inlined]
 [12] deptree_report(::String) at .\REPL[6]:9
 [13] top-level scope at none:0
10

That to me looks like it might be a bug in Base
Base uses regex for some file path operations (like splitdrive),
and if nothing else, that is a leaky abstraction right there.

Maybe MWE it and report on JuliaLang/julia github?

11

I think it’s actually an issue with the script:

Pkg.dir()

is deprecated and returns nothing, which leads to that confusing error message. Here is a fix for the script for Julia 1.0, using pathof:

using Pkg

const _tested = Dict{String, Bool}()

function passing_all_tests(pkgname)
	get!(_tested, pkgname) do
		try
			Pkg.test(pkgname)
			return true
		catch
			return false
		end
	end
end

function get_deps(pkgname)
    Pkg.add(pkgname)
    eval(Expr(:import, Symbol(pkgname)))
    pkg_dir = eval(Expr(:call, :pathof, Symbol(pkgname)))
        @show (pkgname, pkg_dir)
        reqfile = joinpath(pkg_dir, "REQUIRE")
        if !isfile(reqfile)
                # anything without a REQUIRE file is not going to be a problem anyway.
                #This mostly occurs if you are testing something in stdlib
                return String[]
        end
        deps = String[]
        for line in eachline(reqfile)
                (strip(line)=="" || line[1]=='#' || line[1]=='@') && continue
                #^ Skip comments and things like @osx

                pkg_name = split(line)[1]
                pkg_name == "julia" && continue
                push!(deps, pkg_name)
        end
        deps
end

function check(pkgname, final_output=stdout, depth=0)
	if !passing_all_tests(pkgname)
		println(final_output, "\t"^depth * pkgname)
		# don't both checking dependencies of things that pass
		
		# Can not use the below with an asyncmap, or it gets weird
		for depname in get_deps(pkgname)
			check(depname, final_output, depth+1)
		end
	end
end


function deptree_report(pkgname)
	final_output = IOBuffer()
	println(final_output)
	println(final_output, "-"^70)
	println(final_output, "Failing Dependency Tree  for " * pkgname)
	println(final_output, "-"^70)
	
	
	check(pkgname, final_output)
	
	println(final_output, "-"^70)
	println(String(take!(final_output)))
end

(The change here is):

Pkg.add(pkgname)
eval(Expr(:import, Symbol(pkgname)))
pkg_dir = eval(Expr(:call, :pathof, Symbol(pkgname)))
2 Likes