Please refer to the end of: [1909.02659] Automatic Differentiation for Complex Valued SVD
It says that the part of the complex SVD back propagation formula that is unique to the complex case is zero if your loss function only depends on S (in fact it has to depend on both U and V to test the complex back propagation formula). Perhaps try a loss function that depends on U and V like the one they suggest in that paper.