Tests with coverage in VSCode

General Usage
,
1

I recently noticed for the first time this icon in the Julia extension for testing in VSCode:

image
image847×152 7.17 KB

The hover tool tip says “run tests with coverage”. I ran it and got a message saying this feature only works in Julia v 1.11. Installed 1.11 and ran it again - it generated coverage information for a few files then hung.

For the few files with coverage information it was great - I could see the coverage data in the VSCode editor without having to do any manual processing of coverage files.

Will this feature work when 1.11 releases? It would be super handy and convenient.

1 Like
2

Note that there’s “Run Test task” → “Julia: run tests with coverage”. It has been available for years, works for any Julia version, and also displays the results directly in VS Code.

Not sure what the new feature does, but I think it only works on Julia 1.12 and that version won’t be officially released for many months from now – even 1.11 is still not released.

3

I wasn’t aware of that! Is it a VSCode feature or a Julia extension feature? Any links to a doc?

2 Likes
4

Tried “Run Test Task…” on my FastDifferentiation package. It didn’t work for me. Maybe some configuration needs to be set up. I looked at the VSCode docs on this subject but they are long and complicated. Didn’t see how to fix it.

Got this error:

ERROR: LoadError: The following package names could not be resolved:
 * FastDifferentiation.jl (not found in project or manifest)
   Suggestions:
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types C:\Users\seatt\.julia\juliaup\julia-1.11.0-rc2+0.x64.w64.mingw32\share\julia\stdlib\v1.11\Pkg\src\Types.jl:71
 [2] ensure_resolved(ctx::Pkg.Types.Context, manifest::Pkg.Types.Manifest, pkgs::Vector{Pkg.Types.PackageSpec}; registry::Bool)
   @ Pkg.Types C:\Users\seatt\.julia\juliaup\julia-1.11.0-rc2+0.x64.w64.mingw32\share\julia\stdlib\v1.11\Pkg\src\Types.jl:1043
 [3] ensure_resolved
   @ C:\Users\seatt\.julia\juliaup\julia-1.11.0-rc2+0.x64.w64.mingw32\share\julia\stdlib\v1.11\Pkg\src\Types.jl:994 [inlined]
 [4] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{io::IOContext{IO}})
   @ Pkg.API C:\Users\seatt\.julia\juliaup\julia-1.11.0-rc2+0.x64.w64.mingw32\share\julia\stdlib\v1.11\Pkg\src\API.jl:473
 [5] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{coverage::Bool})
   @ Pkg.API C:\Users\seatt\.julia\juliaup\julia-1.11.0-rc2+0.x64.w64.mingw32\share\julia\stdlib\v1.11\Pkg\src\API.jl:159
 [6] test
   @ C:\Users\seatt\.julia\juliaup\julia-1.11.0-rc2+0.x64.w64.mingw32\share\julia\stdlib\v1.11\Pkg\src\API.jl:147 [inlined]
 [7] #test#74
   @ C:\Users\seatt\.julia\juliaup\julia-1.11.0-rc2+0.x64.w64.mingw32\share\julia\stdlib\v1.11\Pkg\src\API.jl:146 [inlined]
 [8] top-level scope
   @ c:\Users\seatt\.vscode\extensions\julialang.language-julia-1.105.2\scripts\tasks\task_test.jl:8
in expression starting at c:\Users\seatt\.vscode\extensions\julialang.language-julia-1.105.2\scripts\tasks\task_test.jl:8

 *  The terminal process "C:\Users\seatt\AppData\Local\Microsoft\WindowsApps\julia.exe '--color=yes', '--project=c:\Users\seatt\Source\FastDifferentiation.jl', 'c:\Users\seatt\.vscode\extensions\julialang.language-julia-1.105.2\scripts\tasks\task_test.jl', 'c:\Users\seatt\Source\FastDifferentiation.jl', 'true'" terminated with exit code: 1. 
 *  Terminal will be reused by tasks, press any key to close it.
5

I think it’s sensitive to the folder name: should be FastDifferentiation, not FastDifferentiation.jl.

6

This is the command line that is executed by Task:Run Test Task on the FastDifferentiation.jl project:

julia.exe '--color=yes', '--project=c:\Users\seatt\source\FastDifferentiation.jl', 'c:\Users\seatt\.vscode\extensions\julialang.language-julia-1.105.2\scripts\tasks\task_test.jl', 'c:\Users\seatt\source\FastDifferentiation.jl', 'true'

and this does not execute properly.

I looked at the task-test.jl file and the problem is clear there. Here are the relevant lines:

path_to_test = ARGS[1]
delete_cov_files = ARGS[2]

package_name_to_run = basename(path_to_test)

using Pkg

Pkg.test(package_name_to_run, coverage=true)

If you run basename on the path to FastDifferentiation.jl you get this:

julia> basename("c:/Users/seatt/source/FastDifferentiation.jl")
"FastDifferentiation.jl"

when what you need is the package name FastDifferentiation.

If your github project is named “*.jl”, and many (most?) are, this won’t work (adding that .jl suffix fantastically improves online search specificity so I don’t think it would be good to recommend that people not do it).

As a test I cloned Symbolics.jl and ForwardDiff.jl and got exactly the same error.

7

Not sure how github is relevant here at all. This VSCode feature is not concerned about github, only about your local folders. And whenever you do ]dev MyPackage in Julia, it creates a folder ~/.julia/dev/MyPackage (not MyPackage.jl), just as VSCode expects.

8

Different working styles I guess. I don’t use ] dev. I use the git integration built into VSCode and just clone the project from github - when you do this the directory the code clones into is named *.jl. Or rather, it’s named exactly the name you gave it on github.

I’ve never understood the advantage of dev so haven’t used it. Everything seems to work fine without it, except for this test task which I didn’t know about till a few days ago.

Are there other features that “just work” if you use dev mode but don’t work if you just clone from github?

9

I’m not familiar with vscode/julia extension internals here, maybe @davidanthoff can chime in (: