The short version is “it depends”. For some problems something discrete is a better inductive bias. For some problems something continuous is a better inductive bias.
c.f. Chapter 1 of On Neural Differential Equations for a bit of this kind of summary. (+Chapter 3 for connections to RNNs.)