I haven’t read the entire discussion, but for differentiating linear algebra routines, there is the package
which provides pure Julia implementations of everything, so that Zygote can differentiate it.
See for example: Does svd() not support forwarddiff? - #3 by maxkapur