Reduce compilation time by avoiding specialization

Would the inferencebarrier trick described in https://github.com/JuliaLang/julia/pull/41931#issuecomment-902545562 and Why doesn't `@nospecialize` work in this example? - #6 by tim.holy work?