Indeed, the problem boils down to having a vector typed with an abstract type, in this case. In some situations that can be avoided by using a tuple (if the vector is small). In other cases that is inherently difficult to solve (operating on containers of abstract types is the hardest thing to optimize, probably), and require sometimes rethinking the structure of the problem to avoid those containers at all.
One of these threads can lead you to long discussions about this last kind of problem: Performance drawback with subtyping
(there is no easy and general solution)