I’ll admit that I’ve never really thought too much about precompilation - I frequently find it surprising when some unrelated package (mostly Plots!) gets precompiled again after some Pkg operation, but I have a beefier laptop so don’t mind the occasional 1-2 minute precompilation breaks.
Therefore I’m not sure whether the following suggestion actually works, but short of compiling a system image, could you just install Plots into the main environment and not change that? As environments are stacked, this allows you to use Plots when working in other project environments, and it shouldn’t be affected by changes to those environments. If you actively use your default environment and frequently change it, maybe you could have a separate Plots environment which you add to the LOAD_PATH with only Plots in which you never (or as you say very infrequently) update?