Sorry, you’re quite right — the basic problem is that the nullspace of M-I is not orthogonal to the other Jordan vectors of M, so the correct projection is non-orthogonal.
I think what you need, instead, is to project using left and right nullspaces. I came up with the following a bit hastily, but it looks right to me and works for your example:
function Minf(M)
R = nullspace(M-I)
L = nullspace(M'-I)'
return R / (L*R) * L
end
You might want to check it a bit more carefully.