Running package tests really slow

General Usage
1

I’m a bit surprised that running unit tests for a simple trial package takes a lot fo time. I am wondering what I’m doing wrong. running

] test

in this example takes 36 seconds. Note that there is no computation involved whatsoever:

# trial/src/trial.jl
module trial

using JSON
using NLsolve
using FastGaussQuadrature
using Plots
using LaTeXStrings
using Interact
using DataFrames

end

# trial/test/runtests.jl
using trial
using Test

@testset "trial.jl" begin
    # Write your tests here.
    @test true
end

the manifest is huge, as a result of the packages used. is there anything I could do to speed this up?

doing a precompile or running this multiple times does not change the result. I thought we don’t have to recompile a bunch of packages upon running each test?

julia> @time Pkg.test()
    Testing trial
Status `/private/var/folders/d2/hbd_ncyd6rg7jj25j7d4ht3h0000gp/T/jl_WAb8sz/Project.toml`
  [a93c6f00] DataFrames v0.21.7
  [442a2c76] FastGaussQuadrature v0.4.2
  [c601a237] Interact v0.10.3
  [682c06a0] JSON v0.21.1
  [b964fa9f] LaTeXStrings v1.1.0
  [2774e3e8] NLsolve v4.4.1
  [91a5bcdd] Plots v1.6.4
  [2ec6069d] trial v0.1.0 `~/ptest/trial`
  [8dfed614] Test
Status `/private/var/folders/d2/hbd_ncyd6rg7jj25j7d4ht3h0000gp/T/jl_WAb8sz/Manifest.toml`
  [79e6a3ab] Adapt v2.0.2
  [4fba245c] ArrayInterface v2.12.0
  [bf4720bc] AssetRegistry v0.1.0
  [6e34b625] Bzip2_jll v1.0.6+4
  [70588ee8] CSSUtil v0.1.1
  [324d7699] CategoricalArrays v0.8.2
  [35d6a980] ColorSchemes v3.9.0
  [3da002f7] ColorTypes v0.10.9
  [5ae59095] Colors v0.12.4
  [bbf7d656] CommonSubexpressions v0.3.0
  [34da2185] Compat v3.16.0
  [e66e0078] CompilerSupportLibraries_jll v0.3.3+0
  [d38c429a] Contour v0.5.5
  [9a962f9c] DataAPI v1.3.0
  [a93c6f00] DataFrames v0.21.7
  [864edb3b] DataStructures v0.18.5
  [e2d170a0] DataValueInterfaces v1.0.0
  [163ba53b] DiffResults v1.0.2
  [b552c78f] DiffRules v1.0.1
  [b4f34e82] Distances v0.9.0
  [5ae413db] EarCut_jll v2.1.5+0
  [c87230d0] FFMPEG v0.4.0
  [b22a6f82] FFMPEG_jll v4.3.1+2
  [442a2c76] FastGaussQuadrature v0.4.2
  [6a86dc24] FiniteDiff v2.6.0
  [53c48c17] FixedPointNumbers v0.8.4
  [f6369f11] ForwardDiff v0.10.12
  [d7e528f0] FreeType2_jll v2.10.1+4
  [559328eb] FriBidi_jll v1.0.5+5
  [de31a74c] FunctionalCollections v0.5.0
  [28b8d3ca] GR v0.52.0
  [5c1252a2] GeometryBasics v0.3.1
  [4d00f742] GeometryTypes v0.8.3
  [cd3eb016] HTTP v0.8.17
  [83e8ac13] IniFile v0.5.0
  [c601a237] Interact v0.10.3
  [d3863d7c] InteractBase v0.10.5
  [41ab1584] InvertedIndices v1.0.0
  [c8e1da08] IterTools v1.3.0
  [82899510] IteratorInterfaceExtensions v1.0.0
  [97c1335a] JSExpr v0.5.2
  [682c06a0] JSON v0.21.1
  [bcebb21b] Knockout v0.2.4
  [c1c5ebd0] LAME_jll v3.100.0+2
  [b964fa9f] LaTeXStrings v1.1.0
  [dd192d2f] LibVPX_jll v1.9.0+0
  [d3d80556] LineSearches v7.1.0
  [1914dd2f] MacroTools v0.5.5
  [739be429] MbedTLS v1.0.2
  [c8ffd9c3] MbedTLS_jll v2.16.8+0
  [442fdcdd] Measures v0.3.1
  [e1d29d7a] Missings v0.4.4
  [d41bc354] NLSolversBase v7.7.0
  [2774e3e8] NLsolve v4.4.1
  [77ba4419] NaNMath v0.3.4
  [510215fc] Observables v0.3.1
  [e7412a2a] Ogg_jll v1.3.4+1
  [458c3c95] OpenSSL_jll v1.1.1+5
  [efe28fd5] OpenSpecFun_jll v0.5.3+3
  [91d4177d] Opus_jll v1.3.1+2
  [bac558e1] OrderedCollections v1.3.0
  [d96e819e] Parameters v0.12.1
  [69de0a69] Parsers v1.0.10
  [fa939f87] Pidfile v1.2.0
  [ccf2f8ad] PlotThemes v2.0.0
  [995b91a9] PlotUtils v1.0.6
  [91a5bcdd] Plots v1.6.4
  [2dfb63ee] PooledArrays v0.5.3
  [3cdcf5f2] RecipesBase v1.1.0
  [01d81517] RecipesPipeline v0.1.13
  [189a3867] Reexport v0.2.0
  [ae029012] Requires v1.0.2
  [992d4aef] Showoff v0.3.1
  [a2af1166] SortingAlgorithms v0.3.1
  [276daf66] SpecialFunctions v0.10.3
  [90137ffa] StaticArrays v0.12.4
  [2913bbd2] StatsBase v0.33.1
  [09ab397b] StructArrays v0.4.4
  [856f2bd8] StructTypes v1.1.0
  [3783bdb8] TableTraits v1.0.0
  [bd369af6] Tables v1.0.5
  [3a884ed6] UnPack v1.0.2
  [0f1e0344] WebIO v0.8.14
  [104b5d7c] WebSockets v1.5.2
  [cc8bc4a8] Widgets v0.6.2
  [83775a58] Zlib_jll v1.2.11+16
  [0ac62f75] libass_jll v0.14.0+3
  [f638f0a6] libfdk_aac_jll v0.1.6+3
  [f27f6e37] libvorbis_jll v1.3.6+5
  [2ec6069d] trial v0.1.0 `~/ptest/trial`
  [1270edf5] x264_jll v2020.7.14+1
  [dfaa095f] x265_jll v3.0.0+2
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [7b1f6079] FileWatching
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [44cfe95a] Pkg
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [10745b16] Statistics
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
Test Summary: | Pass  Total
trial.jl      |    1      1
    Testing trial tests passed 
 36.619714 seconds (1.02 M allocations: 82.031 MiB, 0.15% gc time)
1 Like
2

plausible:

julia> @time begin
       using JSON
       using NLsolve
       using FastGaussQuadrature
       using Plots
       using LaTeXStrings
       using Interact
       using DataFrames
       end
 11.274437 seconds (23.80 M allocations: 1.309 GiB, 4.18% gc time)

what version of Julia are you on?

3 
julia> versioninfo()
Julia Version 1.5.0
Commit 96786e22cc (2020-08-01 23:44 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, broadwell)

julia> @time begin
              using JSON
              using NLsolve
              using FastGaussQuadrature
              using Plots
              using LaTeXStrings
              using Interact
              using DataFrames
              end
 26.535864 seconds (27.13 M allocations: 1.470 GiB, 3.49% gc time)
4

yeah, given your CPU spec, I’m not too surprised, in 1.6 this should improve, see this blog

1 Like
5

oh, that’s a bummer. impossible to test like that.

6

if you’re testing when developing, use Revise.jl is easier. In a REPL, have a test.jl and includet(test.jl) it, then call a main_test() function over and over maybe, you get the idea.

6 Likes
7

this works surprisingly well! thanks. for all the poor folks as slow as me, here is what i did:

using Revise
includet("test/runtests.jl")

# test/runtests.jl
using Trial
using Test

function t1()
	# stuff that would previoulsy go into a @testset
        # i.e. the test body with @test statements etc
end

then I just do t1() on the REPL and it runs my tests. I change the package code of Trial and it is reflectd in the test outcome. :v:

2 Likes
8

If you’re doing that, make sure you add a call to t1() somewhere within your runtests.jl before committing. Otherwise when someone tries to do pkg> test YourPackage it will just define the t1() function but not actually run it.

9

yes, definitely! ideally i will revert back to doing the good old @testset setup in a plain runtests.jl script. this here is just for local testing and to avoid taking forever on each iteration. thanks!

10

I’d have thought that @jling meant that you can keep a long-running session and repeatedly include(test/runtests.jl), using Revise to track changes to the package you’re developing (which should happen automatically™ as long as you’ve deved the package you’re testing). If most of the time is spent loading/precompiling dependencies then just by keeping the process open you should be able to avoid paying that price every time you run the tests.

You might need to ] activate test; dev .. for that to work though (and not commit the changes to test/Project.toml)

1 Like
11

did you mean include or includet here?

12

include, because you want to execute the code in the tests fresh each time. If you used includet on the tests, they’d only run again if the tests themselves changed, and then only the changed parts.

13

well, the assumption is you have a test_main() in your test.jl and you just run that. (which will only change when tests themselves changes anyways?)

Of course if you’re making changes primarily to another package, your test.jl will have using PkgA and thus covered by Revise’s includet?