StochasticDiff.jl and article on automatic differentiation

Hello

It is fun for me to sit back and watch some of the fascinating discussions that happen in the #autodiff channel on Slack. There is definitely some awesome brainpower going into this field of research.

I finished my PhD more than 16 years ago (UIUC 2002) and have been doing mostly nontechnical work in financial services since then, but I try my best to keep one foot in some research to keep my sanity.

Automatic differentiation is a beautiful subject. I was aware of it during graduate school and I could see how it related to what I was working on, but I never had a chance do any serious research on the subject, but have definitely admired it from afar.

A little over a week ago, I pointed out (on Slack) that you could use automatic differentiation for finite differences, i.e. something like â€śautomatic finite differenceâ€ť and was asking if anyone had worked on that. It might sound odd to use AD for FD, but sometimes your underlying space really is finite and a finite difference is the more appropriate model. The subject of my dissertation was differential geometry on finitary spaces so I have thought about this a little bit

Using the same principles, I pointed out that AD could be used for stochastic differentiation. Although the maths is very similar, I figured the stochastic version would be more interesting to more people, so I wrote up a simple proof of concept in the package:

This package is, by no means, intended to be used by anyone in its current state. It is really just a proof of concept and it does that. It actually works

Writing the package was the easy part, but to document it was trickier. There always seemed to me to be a bit of magic around AD. It is unreasonably effective.

So I looked at my code. Scribbled some notes. And iterated until finally a picture emerged that I was happy with. I spent most of the last 7 days condensing the notation and clarifying things to the point that now, in hindsight, it all seems completely natural and obvious

Anyway, I had great fun and wrote up the details in a blog post (with creative name):

If you are a researcher in this area, it might be a big YAWN for you, but it was fun to write up and it leads to all kinds of avenues for future (or past) work. I say â€śor pastâ€ť because I didnâ€™t do any kind of literature search so for all I know this might all be old news from decades ago and obvious to everyone. If that is the case, I donâ€™t mind. It was just a fun exercise for my aging brain so no harm done

In any case, I wanted to share it in case anyone here finds it interesting. I will probably continue to dabble with AD and write up results here and there, but I feel I am decades behind

Cheers

15 Likes

Excellent article!, im not a mathematician, but Iâ€™m looking for something like those projection maps, for application in thermodynamics. in particular, this sounds a lot like a generalization and the underlying base of legendre transforms. if such a package for legendre transforms would be made in julia, then, thermodynamics can do really a leapfrog over any other software available

1 Like

Thanks @longemen3000

If you have a specific problem in mind, Iâ€™d be happy to help out if I could. I agree. Thermodynamics seems like a natural application of this stuff.