If I’m understanding correctly, Virtual.jl should work. Sumtypes.jl or other similar packages should also perform similarly.
Alternatively, if you sort your vector and have separate for loops for the different observations that might help with vectorization.