# Coverage test is extremely slow on multiple threads

**URL:** <https://discourse.julialang.org/t/coverage-test-is-extremely-slow-on-multiple-threads/40682>\
**Category:** Performance\
**Tags:** testing\
**Created:** [June 3, 2020, 3:47pm UTC](https://discourse.julialang.org/t/coverage-test-is-extremely-slow-on-multiple-threads/40682 "2020-06-03T15:47:56Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![erik-f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erik-f/32/18638_2.png) [@erik-f](https://discourse.julialang.org/u/erik-f)\
**Post date:** [June 3, 2020, 3:47pm UTC](https://discourse.julialang.org/t/coverage-test-is-extremely-slow-on-multiple-threads/40682/1 "2020-06-03T15:47:56Z")

</div>

Running `Pkg.test("MyModule", coverage=true)` on more than one thread is extremely slow.

I measured the time on my machine using 1, 2, 8 and 16 threads:

1 thread: 339 seconds  
2 threads: 3106 seconds  
8 threads: 6401 seconds  
16 threads: 5072 seconds

Why is this taking so much time on multiple threads?  
Running `Pkg.test("MyModule")` without coverage takes approximately the same time for every number of threads (around 2 minutes).

When using all 16 threads of my CPU the load on every thread is at 100% pretty much the whole time.

I am using Julia 1.4.1 on Windows 10.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [June 3, 2020, 4:27pm UTC](https://discourse.julialang.org/t/coverage-test-is-extremely-slow-on-multiple-threads/40682/2 "2020-06-03T16:27:03Z")

</div>

Running with code coverage inserts a lot of loads and stores to increment the coverage counters. From my understanding, all threads load and store to the same location (so it’s racy). One core writing to a cache line that is in the cache for another core will invalidate that cache line and the core has to re-fetch it (which can be expensive) to have cache coherence. Maybe that is what you are seeing. Note that this is just a theory.

---

<div class="post-metadata">

**Author:** ![erik-f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erik-f/32/18638_2.png) [@erik-f](https://discourse.julialang.org/u/erik-f)\
**Post date:** [June 3, 2020, 7:19pm UTC](https://discourse.julialang.org/t/coverage-test-is-extremely-slow-on-multiple-threads/40682/3 "2020-06-03T19:19:46Z")

</div>

Is there a way to make this faster in the future? For example every thread could collect it’s own coverage data and then all data can be merge once in the end.

This would make testing much faster for programs that scale very well with the number of threads.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [June 3, 2020, 7:55pm UTC](https://discourse.julialang.org/t/coverage-test-is-extremely-slow-on-multiple-threads/40682/4 "2020-06-03T19:55:12Z")

</div>

> [@erik-f](#):
>
> For example every thread could collect it’s own coverage data and then all data can be merge once in the end.
> 
> This would make testing much faster for programs that scale very well with the number of threads.

Yes, that is what I am thinking as well and that should be possible. You could open an issue about it to keep track of it.

---

<div class="post-metadata">

**Author:** ![erik-f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erik-f/32/18638_2.png) [@erik-f](https://discourse.julialang.org/u/erik-f)\
**Post date:** [June 3, 2020, 10:16pm UTC](https://discourse.julialang.org/t/coverage-test-is-extremely-slow-on-multiple-threads/40682/5 "2020-06-03T22:16:14Z")

</div>

Where would that issue belong? To the julia project or Pkg.jl or somewhere else?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [June 4, 2020, 9:28am UTC](https://discourse.julialang.org/t/coverage-test-is-extremely-slow-on-multiple-threads/40682/6 "2020-06-04T09:28:27Z")

</div>

Definitely to the Julia project.

---

<div class="post-metadata">

**Author:** ![erik-f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erik-f/32/18638_2.png) [@erik-f](https://discourse.julialang.org/u/erik-f)\
**Post date:** [June 4, 2020, 9:36am UTC](https://discourse.julialang.org/t/coverage-test-is-extremely-slow-on-multiple-threads/40682/7 "2020-06-04T09:36:43Z")

</div>

Alright, thank you very much.

> <https://github.com/JuliaLang/julia/issues/36142>
>
> Running \`Pkg.test("MyModule", coverage=true)\` on more than one thread is extreme…ly slow.
> 
> I measured the time on my machine using 1, 2, 8 and 16 threads:
> 
> 1 thread: 339 seconds
> 2 threads: 3106 seconds
> 8 threads: 6401 seconds
> 16 threads: 5072 seconds
> 
> Running \`Pkg.test("MyModule")\` without coverage takes approximately the same time for every number of threads (around 2 minutes) for this module.
> 
> See also the \[discourse discussion\](https://discourse.julialang.org/t/coverage-test-is-extremely-slow-on-multiple-threads/40682).
> 
> I am using Julia 1.4.1 on Windows 10 and my CPU has 8 cores/16 threads.