To sell more expensive cards. Literally, at one point it wasn’t so bad, but then that cut into the data center GPUs, so Nvidia stopped people from using it for FP64 computing through a driver setting. Everyone found out and disabled that to get back the FP64 compute, so they started destroying that part of the cheaper GPUs and sells the same chips with a really high markup for the FP64 (yes, same exact GPU chip in both the $5000 card and the $500 card). Then people started using FP32 a bunch in ML, so they had to ban those from data centers using licensing. Seriously though, if you didn’t live through this story, it’s mind boggling.
The driver story is mentioned in: https://arrayfire.com/explaining-fp64-performance-on-gpus/
How the Titan Black was the king of FP64 from like 2013 to 2018: Reddit - Dive into anything
The new data center ban: https://www.cnbc.com/2017/12/27/nvidia-limits-data-center-uses-for-geforce-titan-gpus.html
So that’s why FP64 is slow, and why people kept buying the Titan Black on the used market years later.
Everything. You will run out of RAM, and you will change your method so it just barely fits.
Not right now, and it’ll be hard to program for unless you have to multiply 4x4 matrices a lot.