Looks like shrinking is still a todo:
https://github.com/JuliaLang/julia/blob/2c9e051c460dd9700e6814c8e49cc1f119ed8b41/base/dict.jl#L238-L242