How do I get Test Coverage not to mess up @allocated calculations

Tooling Profiling
, ,
1

I have a runtests.jl which does something like this.

ff()=@allocated myfun(vara,varb, varc)
@test ff()=0

This returns true if not running with code coverage and then it fails when testing with code-coverage. But not all the time… sometimes code coverage doesn’t run reliably and doesn’t count the source files in my src directory and just counts the runtest.jl file. Sometimes it does though, but I haven’t figured out how to get it to work reliably. At least on one of the Packages I’m developing. I don’t know why. Perhaps it is intermittently crashing somewhere and just not letting me know.

I am running Julia 1.11.

versioninfo()
Julia Version 1.11.4
Commit 8561cc3d68 (2025-03-10 11:36 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: 12 × Intel(R) Core™ i7-10850H CPU @ 2.70GHz
WORD_SIZE: 64
LLVM: libLLVM-16.0.6 (ORCJIT, skylake)
Threads: 12 default, 0 interactive, 6 GC (on 12 virtual cores)

I was using TestReports.jl on my ci, which is why I started trying to switch to Pkg.test() since it seemed to not generate the allocations and fail… but perhaps it just wasn’t running the coverage.
When I just try to execute include(“test/runtests.jl”) with the julia --code-coverage=user set, it fails on the allocations. I do have a separate Project.toml defined in my test directory. I don’t know if that matters.

Is there a way to construct my ff() function to solve this problem?

Best Regards,
Allan Baker

1 Like
2

See: Tests fail on CI, pass locally - #12 by nsajko

2 Likes
3

Thank you. I searched around and didn’t see this. Sounds like the @allocated is kind of busted. Maybe it’s the coverage that’s busted but I would think the accuracy of correctly calculating allocations lives with the @allocated macro. It seems like perhaps @allocated may not be thoroughly tested for accuracy. Perhaps it needs an “ignore/include coverage flag” since it probably is accurate and detecting the inserted coverage code.

But now I think, what other things does including coverage break. Which gets to the other point in the thread. Run tests with coverage and without. That way you get true performance tested and then the allocations calculated. Sort of. It all seems a bit unrefined still.

I find it very challenging to figure out a way to get accurate allocations by the way. Hopefully the way I did it is ok.

4

Example for a CI script that runs the tests with and without coverage testing. When running without I test for allocations.