Coverage test is extremely slow on multiple threads

General Usage Performance
1

Running Pkg.test("MyModule", coverage=true) on more than one thread is extremely slow.

I measured the time on my machine using 1, 2, 8 and 16 threads:

1 thread: 339 seconds
2 threads: 3106 seconds
8 threads: 6401 seconds
16 threads: 5072 seconds

Why is this taking so much time on multiple threads?
Running Pkg.test("MyModule") without coverage takes approximately the same time for every number of threads (around 2 minutes).

When using all 16 threads of my CPU the load on every thread is at 100% pretty much the whole time.

I am using Julia 1.4.1 on Windows 10.

1 Like
2

Running with code coverage inserts a lot of loads and stores to increment the coverage counters. From my understanding, all threads load and store to the same location (so it’s racy). One core writing to a cache line that is in the cache for another core will invalidate that cache line and the core has to re-fetch it (which can be expensive) to have cache coherence. Maybe that is what you are seeing. Note that this is just a theory.

1 Like
3

Is there a way to make this faster in the future? For example every thread could collect it’s own coverage data and then all data can be merge once in the end.

This would make testing much faster for programs that scale very well with the number of threads.

4

Yes, that is what I am thinking as well and that should be possible. You could open an issue about it to keep track of it.

5

Where would that issue belong? To the julia project or Pkg.jl or somewhere else?

6

Definitely to the Julia project.

7

Alright, thank you very much.

1 Like