Forward Automatic Differentiation & FFTs

Specific Domains Numerics
, ,
1

I need to understand if there is a combination of packages that is able to wdo what I need.
I need to do some FFTs, differentiate through them both in Forward and Backward mode with one of the Julia AD engines (it might be possible to use an engine for Forward and one for backward mode) and use into Turing.jl for some bayesian chains.
Backward mode is not really a problem…but what about Forward? Some of my problems have parameter sizes, and hence I am fine with backward mode, but I still would like to be able to use Forward mode for some smaller problems.
So, the list of desiderata is:

  1. An FFT package that works nicely with…
  2. …a Forward AD engine, which work nicely with Turing (hence ADTypes?)

Any suggestion? In this discussion a solution was proposed, but I’d like to have something not relying on a package that has not been actively mantained for years (and likely does not work with Turing.jl).
Is there a working solution to my problem? Or should I just give up on Forward mode AD?

1 Like
2

There’s pull requests to support ForwardDiff.jl, try on these branches:

2 Likes
3

Thanks! I’ll try those today and I’ll report back…in the meantime, thanks!

If it works, do you have a rough estimate of the timeline for the merging?
Thanks again!

4

I’m busy lecturing and other things right now… I imagine it will take a lot of back-and-forth with maintainers. So unfortunately unless you or someone else takes the lead, probably not anytime soon.

5

If it works, I would be interested.
I might ask some guidance, but I think this will also come from the interactions with the mantainers.
I gave a quick look into the AbstractFFTs package. Which are the big chunks missing? Some unit tests for a few lines, is this right? (Beside what the mantainers might ask).

6

If I recall correctly, there’s no real2real test transforms in AbstractFFTs.jl. So we’d have to make like a DCT to test the remaining lines. This can probably be done via the test FFT on a double length vector but I couldn’t remember the formula when I stopped working on it

1 Like
7

If you need a solution right now, don’t want to write rules yourself, and can give up some speed, you could use GenericFFTs.jl. I haven’t used the package in years, but if memory serves that is also an option that fits your needs.

EDIT: no, I just tried and it does have some type restrictions and ForwardDiff doesn’t work out of the box. My apologies.

1 Like
8

Ok, so basically you are saying it has already been implemented but there is lack of proper testing one specific case.
I’ll look into it.

1 Like
9

Ok,
I quickly wrote a stupid piece of code, trying the type FFTs I need (DHT and REDFT00).
ForwardDiff.jl is working when I do NOT use a plan, with a plan it is failing (although I would be completely satisfied even without the plan). A quick check with FiniteDifferences shows a great agreement.
I’ll now go through

So, my understading is that we need to fix 141, to then use the developed tests into 138.
It will take a bit, but I will get back with some proposed fixes.
Thx again!

2 Likes
10

Btw a simpler solution might be to move these extensions to ForwardDiff.jl, as then we can use FFTW.jl to test code in the AbstractFFTs.jl extension.

11

There’s no universal truth as to where package extensions belong, but the answer is usually in the less universal or less widely-used package. Which here means I’d probably aim to put ForwardDiff extensions in the FFT packages, not the other way around.

12

Saying that automatic differentiation is more used than FFTs would shock 99% of all of scientific computing / Matlab users :joy:

2 Likes
13

Indeed, but speaking only about Julia packages, saying that ForwardDiff.jl is used more widely than FFTW.jl is supported (albeit barely) by juliahub.com statistics :slight_smile: