TinyGrad looks like an ML system targeted exclusively at NN models, not a general-purpose AD engine? If you read the source code of TinyGrad, it looks like it has hard-coded derivative rules for a relatively small set of building-blocks (and compositions thereof), with users expected to implement other rules themselves (e.g. in their examples). This is much, much easier than something like Enzyme that aims to AD arbitrary code.
12 Likes