Compile time slowdown from 1.1 to 1.2

Internals & Design
#1

I just tried to benchmark a suite of plots in 1.1.1 and then in 1.2.0 (Windows 64). It includes line plots, scatters, bars, contours and surfaces.

(The underlying notebook is at https://github.com/PaulSoderlind/JuliaTutorial/blob/master/Tutorial_04_Plots.ipynb, but you can easily download it as a .jl file to run it in the REPL).

Julia 1.1.1: 70 seconds
Julia 1.2.0: 150 seconds

(on a half-decent notebook).

EDIT:
I have now updated that notebook to use a slightly different plot() syntax that seems to avoid the 1.2.0 performance penalty. Basically, avoiding title!() etc.

Roadmap for a faster time-to-first-plot?
#2

For my application, Julia 1.1 needs 41 s and Julia 1.2-RC3 51 s (second run from command line).
Julia 1.1 with the option -O0 needs 33s. :slight_smile:

#3

I find travis a fairly (if unreliable and unscientific) indicator here. There apparently is a slow down in 1.2 for some packages:

https://travis-ci.org/JuliaApproximation/ApproxFunOrthogonalPolynomials.jl
https://travis-ci.org/JuliaApproximation/ApproxFunFourier.jl
https://travis-ci.org/JuliaMatrices/BandedMatrices.jl

and a marginal increase in speed in some:

https://travis-ci.org/JuliaApproximation/ApproxFunOrthogonalPolynomials.jl

But none of the slowdowns look like 2x you get (no I didn’t try your example).

Since compile time is a significant amount of Travis timing, it’s disappointing that the claimed improvement on this aspect in 1.2 is not apparent, but the slowdown is reasonable. Anyways, I guess its Julia 1.4 FTW!

#4

I hope you don’t mind I’ve split this out from the meandering TTFP thread. This seems worthy of a more focused discussion.

2 Likes
#5

I’ve been measuring precompilation time in Travis. There seems to be a slight improvement in 1.2.

I checked a few more examples and they had similar trend.

#6

I don’t think there was much work on compiler latency for 1.2 since the people that usually work on that was working on multi threading for most of the release.

4 Likes
#7

I’ve tried to narrow down the problem with the slow plots (from Plots.jl) in 1.2.0. After some trial and error I noticed that this works equally fast in 1.1.1 and 1.2.0 (Win 10):

using Plots
gr()
x = -3:0.01:3
plot(x,cos.(x))
plot!(x,sin.(x))

However, adding the lines

title!("plot 1")
xlabel!("x")
ylabel!("function values")

almost doubles the time in 1.2.0 (but certainly not in 1.1.1). Using pyplot() instead of gr() gives the same general result. I’ll file an issue at Plots.jl.

EDIT:
it seems as if using the syntax

plot(x,cos.(x),
title = "plot 1",
xlabel="x",
ylabel= "function values")
plot!(x,sin.(x))

avoids the 1.2.0 performance penalty. That is, avoid title!, xlabel! and ylabel!

I filed this to Plots.jl. (And no, I do not know the underlying reason for this.)

3 Likes
#8

Simple test (numbers of second execution from bash shown):

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.2.0 (2019-08-20)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> @time using PyPlot
  5.808863 seconds (12.36 M allocations: 620.836 MiB, 6.86% gc time)

julia>

And with Julia 1.1 :

julia> @time using PyPlot
  5.167724 seconds (10.21 M allocations: 524.948 MiB, 4.05% gc time)

So about 11% advantage for Julia 1.1.

#9

I’m a fan of your packages. One thing I don’t understand though is that BandedMatrices is quite slow to load( using BandedMatrices takes 6s on my computer). Do you understand why? Is this because of some sort of design pattern that is hard on Julia right now?

#10

To be honest I’m not sure. Half of the time is from other packages BandedMatrices.jl depends on:

julia> @time using LazyArrays
  2.071962 seconds (3.63 M allocations: 317.976 MiB, 4.92% gc time)

julia> @time using MatrixFactorizations
  0.974763 seconds (2.17 M allocations: 120.067 MiB, 3.04% gc time)

julia> @time using FillArrays
  0.758794 seconds (964.94 k allocations: 46.056 MiB, 1.23% gc time)

julia> @time using BandedMatrices
  3.281776 seconds (21.11 M allocations: 1.304 GiB, 4.56% gc time)

But the 1.3GiB indicates something is going wrong. I’ve managed to fix it in the past by bisection so may try that again.

1 Like
#11

Adding another data point: MathOptInterface (the layer underneath JuMP) has also seen a significant slow down (23min for tests vs 15min): https://travis-ci.org/JuliaOpt/MathOptInterface.jl/builds/575028355.

The difference is after a hacky fix for this wonderful compiler hang: https://github.com/JuliaLang/julia/issues/32167.

Here’s a related and particularly interesting “feature” https://github.com/JuliaOpt/MathOptInterface.jl/pull/834#issuecomment-523641110

#12

Julia 1.1.1: 70 seconds
Julia 1.2.0: 150 seconds

On Julia 1.3.0 and Plots 0.28.1, I now get 35 seconds. Very nice.

3 Likes
#13

Just an update, we’ve narrowed this down to being the method invalidation. Here’s a thread about it:

2 Likes