# Running package tests really slow

**URL:** <https://discourse.julialang.org/t/running-package-tests-really-slow/46587>\
**Category:** General Usage\
**Tags:** package\
**Created:** [September 14, 2020, 1:58pm UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587 "2020-09-14T13:58:07Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 14, 2020, 1:58pm UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587/1 "2020-09-14T13:58:07Z")

</div>

I’m a bit surprised that running unit tests for a simple trial package takes a lot fo time. I am wondering what I’m doing wrong. running

```julia
] test

```

in this example takes 36 seconds. Note that there is _no_ computation involved whatsoever:

```julia
# trial/src/trial.jl
module trial

using JSON
using NLsolve
using FastGaussQuadrature
using Plots
using LaTeXStrings
using Interact
using DataFrames

end

# trial/test/runtests.jl
using trial
using Test

@testset "trial.jl" begin
    # Write your tests here.
    @test true
end

```

the manifest is huge, as a result of the packages used. is there anything I could do to speed this up?

doing a `precompile` or running this multiple times does not change the result. I thought we don’t have to recompile a bunch of packages upon running each test?

```julia
julia> @time Pkg.test()
    Testing trial
Status `/private/var/folders/d2/hbd_ncyd6rg7jj25j7d4ht3h0000gp/T/jl_WAb8sz/Project.toml`
  [a93c6f00] DataFrames v0.21.7
  [442a2c76] FastGaussQuadrature v0.4.2
  [c601a237] Interact v0.10.3
  [682c06a0] JSON v0.21.1
  [b964fa9f] LaTeXStrings v1.1.0
  [2774e3e8] NLsolve v4.4.1
  [91a5bcdd] Plots v1.6.4
  [2ec6069d] trial v0.1.0 `~/ptest/trial`
  [8dfed614] Test
Status `/private/var/folders/d2/hbd_ncyd6rg7jj25j7d4ht3h0000gp/T/jl_WAb8sz/Manifest.toml`
  [79e6a3ab] Adapt v2.0.2
  [4fba245c] ArrayInterface v2.12.0
  [bf4720bc] AssetRegistry v0.1.0
  [6e34b625] Bzip2_jll v1.0.6+4
  [70588ee8] CSSUtil v0.1.1
  [324d7699] CategoricalArrays v0.8.2
  [35d6a980] ColorSchemes v3.9.0
  [3da002f7] ColorTypes v0.10.9
  [5ae59095] Colors v0.12.4
  [bbf7d656] CommonSubexpressions v0.3.0
  [34da2185] Compat v3.16.0
  [e66e0078] CompilerSupportLibraries_jll v0.3.3+0
  [d38c429a] Contour v0.5.5
  [9a962f9c] DataAPI v1.3.0
  [a93c6f00] DataFrames v0.21.7
  [864edb3b] DataStructures v0.18.5
  [e2d170a0] DataValueInterfaces v1.0.0
  [163ba53b] DiffResults v1.0.2
  [b552c78f] DiffRules v1.0.1
  [b4f34e82] Distances v0.9.0
  [5ae413db] EarCut_jll v2.1.5+0
  [c87230d0] FFMPEG v0.4.0
  [b22a6f82] FFMPEG_jll v4.3.1+2
  [442a2c76] FastGaussQuadrature v0.4.2
  [6a86dc24] FiniteDiff v2.6.0
  [53c48c17] FixedPointNumbers v0.8.4
  [f6369f11] ForwardDiff v0.10.12
  [d7e528f0] FreeType2_jll v2.10.1+4
  [559328eb] FriBidi_jll v1.0.5+5
  [de31a74c] FunctionalCollections v0.5.0
  [28b8d3ca] GR v0.52.0
  [5c1252a2] GeometryBasics v0.3.1
  [4d00f742] GeometryTypes v0.8.3
  [cd3eb016] HTTP v0.8.17
  [83e8ac13] IniFile v0.5.0
  [c601a237] Interact v0.10.3
  [d3863d7c] InteractBase v0.10.5
  [41ab1584] InvertedIndices v1.0.0
  [c8e1da08] IterTools v1.3.0
  [82899510] IteratorInterfaceExtensions v1.0.0
  [97c1335a] JSExpr v0.5.2
  [682c06a0] JSON v0.21.1
  [bcebb21b] Knockout v0.2.4
  [c1c5ebd0] LAME_jll v3.100.0+2
  [b964fa9f] LaTeXStrings v1.1.0
  [dd192d2f] LibVPX_jll v1.9.0+0
  [d3d80556] LineSearches v7.1.0
  [1914dd2f] MacroTools v0.5.5
  [739be429] MbedTLS v1.0.2
  [c8ffd9c3] MbedTLS_jll v2.16.8+0
  [442fdcdd] Measures v0.3.1
  [e1d29d7a] Missings v0.4.4
  [d41bc354] NLSolversBase v7.7.0
  [2774e3e8] NLsolve v4.4.1
  [77ba4419] NaNMath v0.3.4
  [510215fc] Observables v0.3.1
  [e7412a2a] Ogg_jll v1.3.4+1
  [458c3c95] OpenSSL_jll v1.1.1+5
  [efe28fd5] OpenSpecFun_jll v0.5.3+3
  [91d4177d] Opus_jll v1.3.1+2
  [bac558e1] OrderedCollections v1.3.0
  [d96e819e] Parameters v0.12.1
  [69de0a69] Parsers v1.0.10
  [fa939f87] Pidfile v1.2.0
  [ccf2f8ad] PlotThemes v2.0.0
  [995b91a9] PlotUtils v1.0.6
  [91a5bcdd] Plots v1.6.4
  [2dfb63ee] PooledArrays v0.5.3
  [3cdcf5f2] RecipesBase v1.1.0
  [01d81517] RecipesPipeline v0.1.13
  [189a3867] Reexport v0.2.0
  [ae029012] Requires v1.0.2
  [992d4aef] Showoff v0.3.1
  [a2af1166] SortingAlgorithms v0.3.1
  [276daf66] SpecialFunctions v0.10.3
  [90137ffa] StaticArrays v0.12.4
  [2913bbd2] StatsBase v0.33.1
  [09ab397b] StructArrays v0.4.4
  [856f2bd8] StructTypes v1.1.0
  [3783bdb8] TableTraits v1.0.0
  [bd369af6] Tables v1.0.5
  [3a884ed6] UnPack v1.0.2
  [0f1e0344] WebIO v0.8.14
  [104b5d7c] WebSockets v1.5.2
  [cc8bc4a8] Widgets v0.6.2
  [83775a58] Zlib_jll v1.2.11+16
  [0ac62f75] libass_jll v0.14.0+3
  [f638f0a6] libfdk_aac_jll v0.1.6+3
  [f27f6e37] libvorbis_jll v1.3.6+5
  [2ec6069d] trial v0.1.0 `~/ptest/trial`
  [1270edf5] x264_jll v2020.7.14+1
  [dfaa095f] x265_jll v3.0.0+2
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles
  [8ba89e20] Distributed
  [7b1f6079] FileWatching
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [44cfe95a] Pkg
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays
  [10745b16] Statistics
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
Test Summary: | Pass Total
trial.jl | 1 1
    Testing trial tests passed 
 36.619714 seconds (1.02 M allocations: 82.031 MiB, 0.15% gc time)

```

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [September 14, 2020, 2:11pm UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587/2 "2020-09-14T14:11:12Z")

</div>

plausible:

```julia
julia> @time begin
       using JSON
       using NLsolve
       using FastGaussQuadrature
       using Plots
       using LaTeXStrings
       using Interact
       using DataFrames
       end
 11.274437 seconds (23.80 M allocations: 1.309 GiB, 4.18% gc time)

```

what version of Julia are you on?

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 14, 2020, 2:12pm UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587/3 "2020-09-14T14:12:17Z")

</div>

```julia
julia> versioninfo()
Julia Version 1.5.0
Commit 96786e22cc (2020-08-01 23:44 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, broadwell)

```

```julia
julia> @time begin
              using JSON
              using NLsolve
              using FastGaussQuadrature
              using Plots
              using LaTeXStrings
              using Interact
              using DataFrames
              end
 26.535864 seconds (27.13 M allocations: 1.470 GiB, 3.49% gc time)

```

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [September 14, 2020, 2:13pm UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587/4 "2020-09-14T14:13:48Z")

</div>

yeah, given your CPU spec, I’m not too surprised, in 1.6 this should improve, see this [blog](https://julialang.org/blog/2020/08/invalidations/)

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 14, 2020, 2:15pm UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587/5 "2020-09-14T14:15:14Z")

</div>

oh, that’s a bummer. impossible to test like that.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [September 14, 2020, 2:15pm UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587/6 "2020-09-14T14:15:32Z")

</div>

if you’re testing when developing, use `Revise.jl` is easier. In a REPL, have a `test.jl` and `includet(test.jl)` it, then call a `main_test()` function over and over maybe, you get the idea.

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 14, 2020, 2:37pm UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587/7 "2020-09-14T14:37:25Z")

</div>

this works surprisingly well! thanks. for all the poor folks as slow as me, here is what i did:

```julia
using Revise
includet("test/runtests.jl")

# test/runtests.jl
using Trial
using Test

function t1()
	# stuff that would previoulsy go into a @testset
        # i.e. the test body with @test statements etc
end

```

then I just do `t1()` on the REPL and it runs my tests. I change the package code of `Trial` and it is reflectd in the test outcome. ✌

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [September 14, 2020, 6:11pm UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587/8 "2020-09-14T18:11:02Z")

</div>

If you’re doing that, make sure you add a call to `t1()` somewhere within your `runtests.jl` before committing. Otherwise when someone tries to do `pkg> test YourPackage` it will just _define_ the `t1()` function but not actually run it.

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 14, 2020, 7:24pm UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587/9 "2020-09-14T19:24:06Z")

</div>

yes, definitely! ideally i will revert back to doing the good old `@testset` setup in a plain `runtests.jl` script. this here is just for local testing and to avoid taking forever on each iteration. thanks!

---

<div class="post-metadata">

**Author:** ![dave.f.kleinschmidt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dave.f.kleinschmidt/32/55_2.png) [@dave.f.kleinschmidt](https://discourse.julialang.org/u/dave.f.kleinschmidt)\
**Post date:** [September 15, 2020, 12:37am UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587/10 "2020-09-15T00:37:30Z")

</div>

I’d have thought that @jling meant that you can keep a long-running session and repeatedly `include(test/runtests.jl)`, using Revise to track changes to the package you’re developing (which should happen automatically™ as long as you’ve `dev`ed the package you’re testing). If most of the time is spent loading/precompiling dependencies then just by keeping the process open you should be able to avoid paying that price every time you run the tests.

You might need to `] activate test; dev ..` for that to work though (and not commit the changes to `test/Project.toml`)

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [September 20, 2020, 11:30am UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587/11 "2020-09-20T11:30:03Z")

</div>

did you mean `include` or `includet` here?

---

<div class="post-metadata">

**Author:** ![dave.f.kleinschmidt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dave.f.kleinschmidt/32/55_2.png) [@dave.f.kleinschmidt](https://discourse.julialang.org/u/dave.f.kleinschmidt)\
**Post date:** [September 20, 2020, 2:32pm UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587/12 "2020-09-20T14:32:22Z")

</div>

`include`, because you want to execute the code in the tests fresh each time. If you used `includet` on the tests, they’d only run again if the tests themselves changed, and then only the changed parts.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [September 20, 2020, 2:43pm UTC](https://discourse.julialang.org/t/running-package-tests-really-slow/46587/13 "2020-09-20T14:43:01Z")

</div>

well, the assumption is you have a `test_main()` in your `test.jl` and you just run that. (which will only change when tests themselves changes anyways?)

Of course if you’re making changes primarily to another package, your `test.jl` will have `using PkgA` and thus covered by Revise’s `includet`?