Upgrading our package from Julia 0.6 to 0.7, we’ve noticed that unit tests use a lot more memory. Trivial tests that should in theory allocate a few KB can now allocate hundreds of MB. A minimal example:
function test() A = [x for x=0.0:2] end test()
julia> @time include("test.jl") 0.236493 seconds (730.15 k allocations: 38.490 MiB, 2.46% gc time)
I understand that the usual answer to this type of question is to time the code twice, since the first run includes JIT, etc. And indeed, if I run
@time test() twice, the second run gives a more reasonable
0.000004 seconds (5 allocations: 272 bytes). But it doesn’t seem very practical when running a large number of unit tests to have to run each test twice (some of our tests take quite a bit of time). Is there a recommended way of doing this? Our goals are 1) somewhat accurate performance measurements so that we can detect regressions, and 2) minimize total time spent running test suite.
In particular, we can’t really tell if we’ve regressed since Julia 0.6, since first invocation of code seems to take a lot longer and use much more memory in Julia 0.7 than in 0.6. Same code above with Julia 0.6.4:
julia> @time include("test.jl") 0.111263 seconds (60.98 k allocations: 3.406 MiB)