That shouldn’t happen, as long as you use the same depot (e.g. .julia) where artifacts are shared between them. Unless of course you’re using different versions of CUDA.jl, in which case different versions of the artifacts may be downloaded.
It is possible to opt out of the artifact system entirely by defining JULIA_CUDA_USE_BINARYBUILDER=false in your environment, but I don’t recommend that. It’s better to try and figure out why the artifacts are being downloaded again.