Discussion in Taking TTFX seriously: Can we make common packages faster to load and use - #9 by jlapeyre seems to answer some of my questions.
Julia bugs to track if you have such problems:
https://github.com/JuliaLang/julia/issues/35972
Also check out the bugs in other packages that link to the ones above.