You are not supposed to remove the ~/.julia folder upon upgrading Julia, because you will also remove all of your environments, (~/.julia/environments) which hold your downloaded packages, too.
You should take a look at this GitHub - JuliaLang/juliaup: Julia installer and version multiplexer for managing versions with Julia.
IIRC you really can’t get around on re-instantiating (and precompiling) packages if you upgrade your julia version.
And if you suffer from long upgrade times due to having lots of packages in your global environment, it is recommended to migrate your workflow to use separate environments with only the relevant packages.