What version of Julia are you running? This is slated to improve significantly — by upwards of 6x on some platforms — on Julia v1.13 with julia#58267. I should check if that’s backport-able to v1.12.
On my machine, v1.11 takes 16.732 ms, whereas the nightly is 3.437 ms.