What lessons could Julia's autodiff ecosystem learn from Stan's TinyGrad?

[ParadaCarleton]

Partly, but most of these operations really are just uncommon–there’s a reason most people can use ReverseDiff.jl just fine without any issues.

[mariusd]

@ToucheSir

But since we’re sharing morals for this particular story, here’s one: before replacing existing AD libraries with a new one, make sure the new one can actually replace them [2]. Even now, the majority of people using NNs would be better off using Tracker.jl or Autograd.jl.

I do very much agree with this.
What currently happens is that most AD effort is put into Diffractor.jl and into maintaining Zygote. These two packages are super-hard to implement and maintain. Just look into the Diffractor’s implementation and you will see why. Same for Enzyme.jl. Few people can deal with them and developing only these packages is not a good sign at all.
What I wish would happen from now on is to focus/promote the simpler packages Tracker.jl or Autograd.jl, which are a lot easier to implement and understand (cost effective) and you can get the community involved. How to promote them? Well, having them working with Flux again. And I think one of them to be the default for Flux. As @ToucheSir mentioned, this will serve nearly all needs in “classical” CNNs.