You’re calculating transpose(cor_mat[i , idx .!= i ]), inv( cor_mat[idx .!= i , idx .!= i ]), etc. twice in each loop. Is this necessary?
Also, you should probably use @views, and in general it is faster and more accurate to do A \ M rather than inv(A) * M.