If the test runner provides us better control of what to test (filtering) and/or how to test (out of order execution etc.) then it’s absolutely useful for me. Oh, implementing file level filtering in test/runtests.jl is rather easy (like this) so what I expect is more precise control like -k of pytest (-grep in your example 
)
By the way, it might be better to provide a way to execute the runner by executing test/runtests.jl. Since the test command of Pkg REPL mode executes the file, I believe majority of projects will continue using the file as an entry point.