What lessons could Julia's autodiff ecosystem learn from Stan's TinyGrad?

As for autodiff, I want to really appreciate how ForwardDiff works in Julia! It felt so nice after Python (a few years ago, maybe they have this now with some magic, idk) to be able to just differentiate any function you throw at it.

Yes, it’s forward mode and isn’t suited for large numbers of inputs — but for other usecases it’s straightforward, painless, and totally general.

5 Likes