You can’t use throwto or schedule(_, exception; error = true) on an arbitrary task that is already started Stop/terminate a (sub)task started with @async - #5 by jameson.
If you are curious, see https://github.com/JuliaLang/julia/pull/41270 for exactly when schedule can be called on a task that is already started. But I don’t think it’d be useful for the use case mentioned in the OP.
Probably the only way to do this reliably is to do the benchmark in a child process. (Or use something like Cassette to inject the cancellation token; but it’d change the program that is benchmarked.)