For all practical intents, in our experience the location of the Julia depot is usually not a performance-critical factor before you scale to >1000 MPI ranks (at least since Julia v1.7).
@VictorEijkhout You should be able to use centrally managed Preferences.jl and/or environment variables to configure the correct settings for MPI.jl and HDF5.jl. For this, you need to put a centrally managed LocalPreferences.jl file in the users’ JULIA_LOAD_PATH (e.g., by setting the JULIA_LOAD_PATH appropriately in a module file). In case of environment variables, you can just set them directly inside, e.g., the Julia module. The contents of the preferences file or the relevant environment variables can be found in the MPI.jl/HDF5.jl docs (let me know if you have trouble finding the right place).
For our purposes, we have found that maintaining centrally managed package installations in a shared depot is not worth the hassle. Even on systems where users cannot access the internet directly, there are ways around that (see, e.g., this topic).