Hello, Julia users. I’ve recently noticed that when I run the unit tests of my module, a significant amount of time appears to be spent on include statements like include("test/path/to/file.jl") where the included files contains a @testset. For example, when I run @time include("test/path/to/file.jl") it tends to takes a few seconds longer to run than the contained testset which I time with @time @testset begin .... end. I realize that the overhead of include() is probably just something you have to accept when using Julia. However, I’m sure there are also some best practices to help reduce this overhead.
What are some tips and tricks to reduce the overhead of including tests in the runtests.jl file?
Interesting. It is something related to the compilation time which is being measured, but there is indeed something odd. In one simple package of mine, I did the following:
I have a @testset begin .. end block. I copied exactly that block into another file, but left it in the runtests.jl file as well. And added time measures. Thus, my runtests.jl file looks like:
@time @testset "Tests" begin
...
end
@time include("./testset.jl")
and then I changed the order to:
@time include("./testset.jl")
@time @testset "Tests" begin
...
end
The results are:
With @testset before include:
Test Summary: | Pass Total
XLStats.jl | 30 30
1.014827 seconds (2.13 M allocations: 107.128 MiB, 9.04% gc time)
Test Summary: | Pass Total
XLStats.jl | 30 30
0.053127 seconds (11.92 k allocations: 590.078 KiB) # include here
With include before @testset:
Test Summary: | Pass Total
XLStats.jl | 30 30
2.177470 seconds (6.29 M allocations: 319.034 MiB, 1.76% gc time) # include here
Test Summary: | Pass Total
XLStats.jl | 30 30
0.000577 seconds (4.70 k allocations: 140.172 KiB)
The overhead of include is therefore the difference between 0.053127 and 0.000577, which excludes the compilation of the test set functions being called.
But why there is a much greater overhead when using include in the first run of the tests, I don’t know. Why something would take longer and allocate so much more to compile because of being included?
Edit: I don’t see any difference in similar tests when executing directly functions or including files that execute functions. That is something odd related to the test set.