Because as I pointed out in the past it is hard to prove that adding a module will not alter previously compiled code.
That’s what our backedges are for. It will always be possible to load one tiny thing and force recompilation of a huge amount of code. For example, loading a module that does this:
import Base: +
+(x::Int, y::Int) = 0 # surprise!
will invalidate any method that adds Ints, so future calls will have to compile everything they need from scratch. The bigger problem is indeed tricky, but questions like “can we at least cache more of the results of inference? will it help in some circumstances?” are certainly going to have affirmative answers.