Workflow for faster testing of Julia packages?

MilesCranmer · May 4, 2024, 3:40pm

What is the fastest possible way to run a Julia test suite (and also get test coverage)?

Normally I call

julia --startup-file=no --code-coverage=user --project=. -e 'using Pkg; Pkg.test(coverage=true)'
julia --startup-file=no coverage.jl

where coverage.jl is:

using Coverage
# process '*.cov' files
coverage = process_folder() # defaults to src/; alternatively, supply the folder name as argument
push!(coverage, process_folder("ext")...)

LCOV.writefile("lcov.info", coverage)

# process '*.info' files
coverage = merge_coverage_counts(
    coverage,
    filter!(
        let prefixes = (joinpath(pwd(), "src", ""), joinpath(pwd(), "ext", ""))
            c -> any(p -> startswith(c.filename, p), prefixes)
        end,
        LCOV.readfolder("test"),
    ),
)
# Get total coverage for all Julia files
covered_lines, total_lines = get_summary(coverage)
@show covered_lines, total_lines

However, this takes 4 minutes in total to run in its entirety (for the package I’m currently working on^[1]), so tends to slow me down when I am trying to iterate on my test suite – specifically with the goal of improving code coverage.

The other workflow I have tried is to use Revise, and call my test suite from the REPL:

julia> @eval module $(gensym())
           include("test/runtests.jl")
       end

where I put it in a module to avoid issues with test-specific structs changing.

This tends to be faster, but (1) I’m not sure how to get coverage out of this, and (2) it still feels like the tests themselves are not “julia speed” [see more notes below].

Any tips for speeding up this part of the development process? What is the fastest possible way I can compute code coverage for my test suite? Alternatively, is there a way I can dynamically generate code coverage without restarting Julia?

Regarding (2) – perhaps I need to wrap more things in functions, or perhaps the @test and @testset macros should be improved in some way (?).

For example, if you expand the `@testset` macro, there is a lot of dynamic dispatch going on:

julia> @macroexpand @testset "blah" begin
           x = 1
           @test x == 2
       end
quote
    #= REPL[9]:2 =#
    begin
        #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1559 =#
        Test._check_testset(if Test.get_testset_depth() == 0
                Test.DefaultTestSet
            else
                Test.typeof(Test.get_testset())
            end, $(QuoteNode(:(get_testset_depth() == 0))))
        #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1560 =#
        local var"#13375#ret"
        #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1561 =#
        local var"#13371#ts" = if if Test.get_testset_depth() == 0
                                Test.DefaultTestSet
                            else
                                Test.typeof(Test.get_testset())
                            end === Test.DefaultTestSet && true
                    #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1562 =#
                    (if Test.get_testset_depth() == 0
                        Test.DefaultTestSet
                    else
                        Test.typeof(Test.get_testset())
                    end)("blah"; source = Symbol("REPL[9]"), Test.Dict{Test.Symbol, Test.Any}()...)
                else
                    #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1564 =#
                    (if Test.get_testset_depth() == 0
                        Test.DefaultTestSet
                    else
                        Test.typeof(Test.get_testset())
                    end)("blah"; Test.Dict{Test.Symbol, Test.Any}()...)
                end
        #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1566 =#
        Test.push_testset(var"#13371#ts")
        #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1570 =#
        local var"#13372#RNG" = Test.default_rng()
        #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1571 =#
        local var"#13373#oldrng" = Test.copy(var"#13372#RNG")
        #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1572 =#
        local var"#13374#oldseed" = (Test.Random).GLOBAL_SEED
        #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1573 =#
        try
            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1575 =#
            (Test.Random).seed!((Test.Random).GLOBAL_SEED)
            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1576 =#
            let
                #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 =#
                begin
                    #= REPL[9]:2 =#
                    x = 1
                    #= REPL[9]:3 =#
                    begin
                        #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:508 =#
                        if false
                            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:509 =#
                            Test.record(Test.get_testset(), Test.Broken(:skipped, $(QuoteNode(:(x == 2)))))
                        else
                            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:511 =#
                            let var"#13378#_do" = if false
                                        Test.do_broken_test
                                    else
                                        Test.do_test
                                    end
                                #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:512 =#
                                var"#13378#_do"(begin
                                        #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:668 =#
                                        try
                                            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:669 =#
                                            Test.eval_test(Test.Expr(:comparison, x, ==, 2), Test.Expr(:comparison, :x, :(==), $(QuoteNode(2))), $(QuoteNode(:(#= REPL[9]:3 =#))), $(QuoteNode(false)))
                                        catch var"#13380#_e"
                                            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:671 =#
                                            var"#13380#_e" isa Test.InterruptException && Test.rethrow()
                                            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:672 =#
                                            Test.Threw(var"#13380#_e", (Test.Base).current_exceptions(), $(QuoteNode(:(#= REPL[9]:3 =#))))
                                        end
                                    end, $(QuoteNode(:(x == 2))))
                            end
                        end
                    end
                end
            end
        catch var"#13377#err"
            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1580 =#
            var"#13377#err" isa Test.InterruptException && Test.rethrow()
            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1583 =#
            Test.trigger_test_failure_break(var"#13377#err")
            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1584 =#
            if var"#13377#err" isa Test.FailFastError
                #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1585 =#
                if Test.get_testset_depth() > 1
                    Test.rethrow()
                else
                    Test.failfast_print()
                end
            else
                #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1587 =#
                Test.record(var"#13371#ts", Test.Error(:nontest_error, Test.Expr(:tuple), var"#13377#err", (Test.Base).current_exceptions(), $(QuoteNode(:(#= REPL[9]:1 =#)))))
            end
        finally
            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1590 =#
            Test.copy!(var"#13372#RNG", var"#13373#oldrng")
            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1591 =#
            (Test.Random).set_global_seed!(var"#13374#oldseed")
            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1592 =#
            Test.pop_testset()
            #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1593 =#
            var"#13375#ret" = Test.finish(var"#13371#ts")
        end
        #= /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-HL2F7YQ3XH.0/build/default-honeycrisp-HL2F7YQ3XH-0/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/Test/src/Test.jl:1595 =#
        var"#13375#ret"
    end
end

I wonder if the macro could instead statically converting the @tests into function calls or something, instead of eval’ing them at runtime. Most of the recent changes to Test.jl have been bug fixes and documentation rather than performance improvement: (see commits, apart from maybe #42518) so perhaps it could be worth refactoring the code a bit.

If wanting a specific example of my use-case, here’s the test suite I am working on at the moment: DynamicExpressions.jl/test at master · SymbolicML/DynamicExpressions.jl · GitHub. ↩︎

mkitti · May 4, 2024, 3:42pm

Have you seen the TestItems.jl and TestItemRunner.jl packages?

MilesCranmer · May 4, 2024, 3:48pm

I hadn’t, thanks.

At the moment there are two ways to run @testitems: with the integrated test UI in the Julia VS Code extension, or with this package TestItemRunner.jl. In both cases test item detection is based on syntactic analysis, i.e. no code from your package is run to detect test items. Both execution engines will instead look at all *.jl files in your package folder, identify all @testitem calls and then provide ways to run them.

I definitely like the sound of this!

The one thing I don’t see any notes of is code coverage. Do you know if it integrates with Coverage.jl? I would love to just be able to quickly update the relevant .cov files so I can pull in the new changes to Coverage Gutters - Visual Studio Marketplace, rather than needing to re-run the entire suite each time.

Also, does it let me run the tests for only a single module, rather than the entire package?

mkitti · May 4, 2024, 3:52pm

See the following thread?

MilesCranmer · May 4, 2024, 3:55pm

That thread is what my coverage.jl file is from actually! (see coverage.jl file on my first post)

What I’m wondering is whether you can generate the *.cov files from this new test package or not.

MilesCranmer · May 4, 2024, 3:57pm

Just to clarify, my main question is: what is the fastest way to re-generate code coverage from my test suite after I make a code change?

kellertuer · May 4, 2024, 4:05pm

That might also depend a bit on how the test suite is set up, that is, for example, how much the tests are modularised – for example with the already mentioned TestItemRunner and VS Code.

I try to write tests per feature that are “standalone” in the sense that they also would run via include – or in other words each .jl file in test/ loads it necessary packages before running.
(The main file then just includes all these in a large testset)

Example: The file Manopt.jl/test/solvers/test_particle_swarm.jl at master · JuliaManifolds/Manopt.jl · GitHub just runs tests for one of the solvers of that package. So including that with code coverage check activated gives you code coverage for only that test

Remark: We started that on one of our packages, since it was more a library of things – and running all tests took 25 Minutes – but split over 45 “items” in said library, so then we could run them individually.
And probably the TestItem/TestRunner is even a more structured way to this (the linked file with tests is for example older than that pair of packages).

davidanthoff · May 4, 2024, 6:00pm

Adding coverage support in the VS Code UI of the test item framework is something I’d love to do, but that is currently blocked by More fine-grained coverage output control · Issue #50215 · JuliaLang/julia · GitHub.

One can get coverage with the CI runner, but then one doesn’t get any of the benefits of fast turnaround times etc.

MilesCranmer · May 4, 2024, 6:28pm

Thanks for sharing your workflow.

Here’s more context: this is my current test folder, which as you can see, is split up into small files that test a specific feature or module: DynamicExpressions.jl/test at 749385a1183b4ea8cd4145c914b08b6c8d9774c6 · SymbolicML/DynamicExpressions.jl · GitHub

I have this file unittest.jl that uses SafeTestsets.jl to include each file: DynamicExpressions.jl/test/unittest.jl at 749385a1183b4ea8cd4145c914b08b6c8d9774c6 · SymbolicML/DynamicExpressions.jl · GitHub

So in principle I can also just include any of those specific tests in my REPL, and it should run those small tests.

But, in practice, how do you get the coverage for one of those “items”?

Here’s my current workflow with this in mind:

julia --project=. --code-coverage=user

then

julia> using TestEnv; TestEnv.activate()  # Get test deps

julia> @eval module $(gensym())
           using Test
           @testset begin
               include("test/test_parse.jl")
           end
       end

But, no .cov files are generated. It’s only when I quit the REPL… So how can I make those get generated during an active REPL session?

Edit: seems like it only gets generated when it’s quit due to julia/src/coverage.cpp at da6892ffc9fbb9a7ae272f570f3fd6002439de6c · JuliaLang/julia · GitHub being called upon Julia exit, but there being no mechanism for calling it earlier.

MilesCranmer · May 4, 2024, 6:35pm

(Edit: resolved, see next post)

MilesCranmer · May 4, 2024, 6:57pm

Did some digging on More fine-grained coverage output control · Issue #50215 · JuliaLang/julia · GitHub. Seems like I need to call the C function :jl_write_coverage_data to write the coverage data earlier.

Although seems like it is not exported for some reason (?):

julia> ccall((:jl_write_coverage_data, "libjulia"), Cvoid, (Cstring,), "output.info")
ERROR: could not load symbol "jl_write_coverage_data":
dlsym(0x8761f210, jl_write_coverage_data): symbol not found
Stacktrace:
 [1] top-level scope
   @ ./REPL[15]:1

kellertuer · May 4, 2024, 7:20pm

Oh I might be a bit more lazy on that.

I have a config for my tests, i.e. some options in the runtests file that specify which group to run and then use GitHub - JuliaCI/LocalCoverage.jl: Trivial functions for working with coverage for packages locally. locally.

MilesCranmer · May 4, 2024, 7:57pm

This doesn’t help with the actual issue though – the slowness is because I am forced to restart the Julia runtime and re-load the test suite (or a part of it) every time I want to run the test and get code coverage. LocalCoverage doesn’t get around this, it just manages the *.cov files.

But the issue is that those coverage files don’t get generated until Julia exits.

Coverage is only written when the Julia runtime exits: julia/src/init.c at da6892ffc9fbb9a7ae272f570f3fd6002439de6c · JuliaLang/julia · GitHub which is in the jl_atexit_hook function. This seems to be the core issue here.

kellertuer · May 4, 2024, 8:00pm

ah, ok, i understand. For me restarting Julia and the tests was not a problem (timewise) in my workflow, but really the runtime of all tests. Then we have different scenarios.

mkitti · May 4, 2024, 8:22pm

Try this:

julia> ccall(:jl_write_coverage_data, Cvoid, (Cstring,), "output.info")

julia> @ccall jl_write_coverage_data("output.info"::Cstring)::Cvoid

ianshmean · May 4, 2024, 9:22pm

What’s the purpose of specifying --code-coverage=user then Pkg.test(coverage=true)?

You should just need to do Pkg.test(coverage=true) and it will be faster because coverage is only turned on for the package being tested.

MilesCranmer · May 4, 2024, 9:35pm

Thanks! I’ll try again and see whether it can be done…

Thanks, I didn’t realise it wasn’t required. I think I found it in some CI script and have just been doing it that way ever since…

I am trying to avoid Pkg.test since it launches another Julia process (and thus can’t exploit Revise’d methods)

ianshmean · May 4, 2024, 9:53pm

Check out the docs for the --code-coverage arg then, it takes a path now (prefixed with a @). That will be faster than user

MilesCranmer · May 4, 2024, 11:00pm

Thanks!

Okay I think I got everything working. Thanks to everyone for your help.

Here’s my new workflow for getting rapid-fire code coverage updates:

Start Julia with --code-coverage=@ to only track the current package you are developing.

julia \
    --code-coverage=@ \
    --code-coverage=lcov.info \  # To prevent generating the .cov files at exit; otherwise this is not needed
    --depwarn=yes \
    --project=.

Use TestEnv.jl to load up all the test dependencies:

julia> using TestEnv; TestEnv.activate()

Evaluate only the parts of the test suite you want to accumulate coverage for. (I recommend doing this by inclusion in a new module) –

julia> @eval module $(gensym())
           using Test; @testset begin
               include("test/test_parse.jl")
           end
       end;
Test Summary: | Pass  Total  Time
test set      |   54     54  0.6s

Generate an LCOV-compatible coverage file:

julia> @ccall jl_write_coverage_data("lcov.info"::Cstring)::Cvoid

Now, lcov.info will appear in your current directory. If you use coverage gutters in VSCode (highly recommended), it will automatically detect this file and generate interactive code coverage metadata for your entire VSCode project.

After making an edit to your unittests, and re-run (3) and (4) as many times as you want to update the coverage with the aggregate[^1]. It will be fast as you won’t have to restart Julia again!

Happy testing

P.S., if anybody knows the right function call to empty a StringMap< SmallVector<logdata_block*, 0> >, we could also automatically empty the coverage information. Otherwise it will just keep accumulating throughout the runtime.

P.P.S., I am not sure how this interacts with Revise. Maybe [P.S.] is needed for that, otherwise the line info would get thrown off. But maybe it’s fine…

MilesCranmer · May 4, 2024, 11:32pm

Okay, I have added a function that clears the coverage data to permit the workflow described above: Create `jl_clear_coverage_data` to dynamically reset coverage by MilesCranmer · Pull Request #54358 · JuliaLang/julia · GitHub