I’m observing extremely long compilation times of more than 2000 sec on Appveyor (Win32) for the delay differential equation solver package DelayDiffEq.
A week ago, with version 5.7.0 the first test took around 160 sec on Win32 (the test failures are due to upstream issues; the timings are roughly the same as a week before). However, on version 5.8.0 we’re suddenly up to more than 2000 sec.
In version 5.8.0, we introduced a major change under the hood, that consists mainly of an easier and more consistent handling of solutions and a simpler setup of the solvers. Intuitively the simpler setup should also reduce the amount/layers of nested types. In our benchmark we could not observe any performance regressions, and also the times and allocations of all subsequent tests (apart from the first ones) are still similar to version 5.7.0 (and earlier).
@andreasnoack noticed that we’re accessing closed over variables in some closures, which apparently is undefined behaviour, so we fixed that in version 5.9.0 yesterday night. However, the compilation time issues persist.
I also noticed that we have some type inference problems in the initialization of our integrators. Reducing the number of non-inferred parameters improves the performance quite drastically (with respect to times and allocations) in our tests, but still the initial test takes more than 2000 sec to run.
Interestingly, the problems are a lot less severe on Win64 and Linux 64bit (Travis), but still compilation takes too long IMO, especially compared with the time it takes to run all subsequent tests.
Has anyone observed anything like this before? Does anyone have an idea what could cause these extremely long compilation times? And how to profile it?