Half storage? How can you do that? Array{Nothing,1} allocates only 80 bytes even if it contains many elements.
julia> @allocated a = Array{Int}(undef,10000000)
80000080
julia> @allocated a = Array{Nothing}(undef,10000000)
80
The problem here is that you have only 5 elements in this set, but a hash table with n = 16
slots (with 16 UInt). Also, if you check the keys of the set, you will find it contains 16 keys:
julia> set.dict.keys
16-element Vector{Int64}:
So another half time is spent to copy the slots and these additional keys.