Right, let me first thank you all for all this awesome input.
After testing and playing around I think I agree with everybody’s advice:
I’ll improve on the detection of the objects, thus reducing the number of possible leads, as well as, be able to prune unlikely ones. This way I hope to use something a lot simpler to store the tracks (like a binary tree, e.g. LeftChildRightSiblingTrees.jl).
Specifically, I detect an object (read animal) in the next frame depending on the location of the object in the previous frame, so a sense of proximity is already in use. I plan to add a Kalman filter to the mix since the trajectory of the animal doesn’t normally change much in terms of direction and speed. I hope to stay clear from morphological operations and segmentation, feature detection, and neural networks for as long as I can. Right now, I seem to get really far by subtracting the background and the fact the animals are much darker than everything else.