Hmm, just noticed that you need this for 3D (ImageProjectiveGeometry.ransacfithomography is for 2D, as is estimateRigidTransform). Still, the ransac function could be a good start, even though there are some obvious ways in which its performance can be improved.