I think the crux of the issue is that you included transfer time.
If you do have to transfer data to and from the GPU, I think it’s really unlikely for that to be worth it.
However, if you can keep your data on the GPU, doing the vast majority of your calculations there, then it is worthwhile.
Training neural networks is an ideal example of that.
Personally, I have struggled to make good use of GPUs. I still really want to sit down sometime and actually learn how to optimize them, or simply write code that can run on them and stay there.
(This is why I am such a big fan of AVX-512. While it still doesn’t come close to GPU-speed/$, it is a big step up from avx2, and something I’ve actually been capable of using.)