Mac "Resolving package versions..." takes a long time

New to Julia
1

I’ve been using Julia for a while, but this has me flummoxed. At work on Arch, a package test spends minimal time in the package resolving stage, but on my Mac it drags.

I have quite a few dependencies, and I would understand it taking a while the first time…

Really a killer on actually doing development when testing takes an eon to just fail.

2 Likes
2

That is interesting, I have personally never seen it be slow. What time scale are we talking about? What’s the stacktrace from a Ctrl+C? If you copy and paste the environment (Project.toml + Manifest.toml) to another system, is it still slow?

1 Like
3

Same with me on Windows - 5-7 sec on a very basic code and simple tests. Ctrl-C does not reveal much:

(LessOLS) pkg> test
   Testing LessOLS
 Resolving package versions...
ERROR: LoadError: InterruptException:
Stacktrace:
 [1] include(::String) at .\client.jl:392
 [2] top-level scope at none:0
in expression starting at D:\github\LessOLS.jl\test\runtests.jl:2
ERROR: Package LessOLS errored during testing

Also when running tests on Travis - it takes about a centrury (> 1min) to pass after “Resolving package versions” and complete tests at Travis CI - Test and Deploy Your Code with Confidence

Cross-posting at github issues

4

For me I get:

julia> import Pkg

julia> @time Pkg.test("LessOLS")
   Testing LessOLS
 Resolving package versions...
   Testing LessOLS tests passed 
  4.706391 seconds (822.65 k allocations: 47.687 MiB, 0.25% gc time)

Note that the code takes quite a lot of time to precompile:

julia> @time using LessOLS
[ Info: Precompiling LessOLS [3e4c6e70-f926-11e8-0fa5-39915c5a2b72]
 39.171277 seconds (2.73 M allocations: 151.785 MiB, 0.40% gc time)

which is what Travis has to do before the tests are run. The whole process taking ~50 seconds on the Travis nodes doesn’t seem that odd comparing to the time I get on my laptop.

2 Likes
5

Why could the precompile be very long? Or perhaps is it too broad question

6

Seems like Distributions is slow to precompile.

julia> @time using Distributions
[ Info: Precompiling Distributions [31c24e10-a181-5473-b8eb-7969acd0382f]
 36.369072 seconds (2.67 M allocations: 148.948 MiB, 0.36% gc time)
2 Likes
7

I cleaned up the compiled cache by deleteing everything at /.julia /compiled and compile time reduced, here are times before and after:

julia> @time using LessOLS
  1.934703 seconds (1.60 M allocations: 95.501 MiB, 2.34% gc time)

julia> @time using LessOLS
  0.001462 seconds (587 allocations: 31.250 KiB)

Could it be Travis had Julia installation with precompiled cache? I would expect the oppisite though.

8

I don’t believe those numbers, it should print [ Info: Precompiling LessOLS. Did you restart julia in between?

9

You are right, I did measure without exiting the session. After exiting the session the precompile time is similar to what you have:

julia> @time using LessOLS
[ Info: Precompiling LessOLS [3e4c6e70-f926-11e8-0fa5-39915c5a2b72]
 57.375232 seconds (2.92 M allocations: 159.585 MiB, 0.20% gc time)
10

Most of 57 sec time is precompilation of Distributions:

julia> @time using Distributions
[ Info: Precompiling Distributions [31c24e10-a181-5473-b8eb-7969acd0382f]
 50.983350 seconds (2.91 M allocations: 159.384 MiB, 0.21% gc time)
11

I wish there was a way around precompiling dependencies for local tests - waiting for cycles of 10s sec on trivial tests is so discouraging for development.

1 Like
12

You can always just do include("~/.julia/dev/MyPackage/tests/runtest.jl") with Revise.jl to avoid recompiling dependencies after every time you change something.

2 Likes
13

Things are much better with Revise.jl. On a firest run there is still rather long wait time, of 19 sec, but on the second - the tests run really fast.

julia> @time include("test/runtests.jl")
 19.758721 seconds (5.05 M allocations: 262.562 MiB, 3.40% gc time)
Test Passed

julia> @time include("test/runtests.jl")
  0.046278 seconds (4.27 k allocations: 224.748 KiB)
Test Passed

The 19 seconds are precompilation times for Distributions and Statistics packages, among other things.

julia> @time using Distributions
  8.965152 seconds (1.59 M allocations: 94.658 MiB, 2.86% gc time)

julia> @time using Statistics
  6.460659 seconds (1.81 M allocations: 84.953 MiB, 1.20% gc time)

Not sure what changed from previous versions, but this is much better than 50 seconds for compiling Distributions, as it was before.

One thing that still bothers me: pkg"test" will still do all the precompilations and make usual test-driven cycle really impossible due to long wait times. Seems like Revise.jl is a must for TDD in Julia. So why the standard Pkg.test is so heavyweight and anyone wanting to do TDD must discover Revise.jl the hard way in Julia? Maybe there can be option to incorporate “test --no-precompile” to Pkg?

2 Likes