The title pretty much. See this Github workflow for an example. 1.12 and 1.11 take 8 minutes, vs 1.10 at 3 minutes.
Does anyone know why this is happening?
2 Likes
This is indeed a problem, and I’ve noticed it too. But I’m not sure what’s behind it.
1 Like
People have been wondering this for a while, there is even a precompilation etc. benchmark, where timings can vary with setup enough that it’s not as simple as your observed regression at 1.11. For 1.11 specifically, people often bring up moving Array code from C into Julia and stdlib excision from the sysimage. We can expect some changes to compilation from those, but there’s no easy way to tell how much those factor into any specific situation.
I don’t think anybody has really come up with a satisfactory explanation yet. Part of the problem is that varying the Julia version also changes LLVM and what packages are compatible in otherwise identical environments, so that’s a whole lot that changes outside of the base Julia repo and hasn’t been finely controlled in any benchmarks so far. Does anyone know if we can at least list out every method specialization and how long it took to compile each? Maybe we could at least try to compare across version changes.
2 Likes
I think in this package specifically is hurt more than most by the memory changes. I was taking a look and the __generic_kernel! that is the core computational kernels of this function (which were already huge functions) went from around 22k lines of IR in 1.10 to 55k in 1.12. This means that Julia has to do a lot more work here, same for LLVM. So that already would increase the compile time by a lot, and then there is the fact that compilation/optimization is rarely a O(N), it’s usually slightly more than that.
For reference after optimization the LLVM that ends up being generated is 10k lines of IR less in 1.12 over 1.10 (52k vs 40k)
1 Like
One datapoint/tool here is the new julia CI timing tracking site. If you just look at building julia master for CI (various platforms, configurations) over the last year, there have been some temporary blips but it’s trended upwards +33% on average. Building julia !== precompiling packages, and different agents are used so it’s noisy, but it might be helpful for visually identifying jumps.
All build jobs, all time (data only available since end of 2024)
2 Likes