Pkg.test() is orders of magnitude slower than include("test/runtests.jl")

General Usage Performance
1

If I run the following (starting Julia with only --project switch)

include("test/runtests.jl")

@time of my function gives

0.030818 seconds (130.79 k allocations: 7.158 MiB)

But, if I run

using Pkg; Pkg.test(;julia_args=`--check-bounds=no`)

Then the result is:

1.420798 seconds (148.56 k allocations: 8.402 MiB)

Other than --check-bounds, what else could be causing a major slow down?

2

It starts a new Julia process which means it also needs to load any packages, compile functions, etc for the first time (just like when you start a new Julia session).

3

Sorry I should have explained, I’m using @time on my function call (several times to avoid compilation timing and I’m report the last run). So this does not include the time of loading packages, etc. I’m seeing a very different performance of my package during testing vs. directly running my test script.

4

Ah ok, that makes sense. Maybe code coverage or deprecation warnings? The command Pkg uses is here btw: Pkg.jl/Operations.jl at 90b535df2467cade9f1e63c1607fbc0ddf2303a4 · JuliaLang/Pkg.jl · GitHub

5

OK, I found the solution.

I included at the top of runtests.jl the following:

display(Base.JLOptions())

Then I compared the options in test mode and script mode. Turns out depwarn was the culprit.

So now when running:

using Pkg; Pkg.test(;julia_args=`--depwarn=no`)

now gives the same time as running include("test/runtests.jl")