Thought I’d leave this here as well: I wanted to get a feel for how the user experience of the “original” TTFX - i.e. time-to-first-plot using Plots.jl - has evolved over the years, so I wrote a little benchmark script that times starting Julia and running using Plots; display(scatter(rand(1_000))) for all Julia versions since 0.7.
Here’s what the mean and minimum times look like across versions:
Initially I had included 0.6.4 as well but with such an old Julia version one has to also fall back onto a very old Plots version which errors out on the display call. An alternative script with savefig instead of display suggests that 0.6 had similar TTFP to 0.7-1.2.
I think the plot makes clear the scale of achievement here - while I’m sure there’s more to come as the ecosystem starts to make best use of the new abilities of Julia, it seems clear that this is the first time that we’ve seen a dramatic qualitative shift in the sort of very basic experience that the modal user might have without resorting to any tricks like PackageCompiler.
Some more boring details for those interested below:
Details
The benchmarks were run on an i7 Windows laptop, versioninfo() is:
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: 8 × 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-13.0.1 (ORCJIT, tigerlake)
Threads: 1 on 8 virtual cores
As Plots.jl has dropped compatibility for older Julia versions over time, going back in Julia versions also means going back to older Plots versions, so the benchmarks don’t just capture differences in precompilation and other language features, but changes in Plots itself.
The following table shows the Plots version used for every Julia version, as well as the average, minimum, and standard deviation of the 20 samples taken:
│ String String Float64 Float64 Float64
─────┼───────────────────────────────────────────────────
1 │ 0.6.4 0.17.4 11.5 10.7 0.9
2 │ 0.7.0 0.19.3 21.5 19.8 1.3
3 │ 1.0.5 1.4.3 18.5 18.1 0.5
4 │ 1.1.1 1.4.3 21.3 18.3 4.4
5 │ 1.2.0 1.4.4 22.8 21.4 1.5
6 │ 1.3.1 1.6.12 27.3 26.2 0.7
7 │ 1.4.2 1.6.12 35.0 30.0 5.4
8 │ 1.5.4 1.24.3 35.7 31.9 3.3
9 │ 1.6.5 1.27.6 24.6 21.7 2.4
10 │ 1.7.3 1.37.2 24.3 22.6 2.0
11 │ 1.8.3 1.38.0 20.6 17.7 2.8
12 │ 1.9.0b 1.38.0-dev 7.6 4.1 1.1
Note that 1.9.0b uses a branch of Plots with a fix from Kristoffer that addresses an issue with the usage of cached code on Windows.
The whole benchmark code run is:
using DataFrames, ProgressMeter
path = "path/to/Julias"
julias = Dict(
"0.6.4" => "Julia-0.6.4/bin/julia.exe",
"0.7.0" => "Julia-0.7.0/bin/julia.exe",
"1.0.5" => "Julia-1.0.5/bin/julia.exe",
"1.1.1" => "Julia-1.1.1/bin/julia.exe",
"1.2.0" => "Programs/Julia-1.2.0/bin/julia.exe",
"1.3.1" => "Julia-1.3.1/bin/julia.exe",
"1.4.2" => "Programs/Julia/Julia-1.4.2/bin/julia.exe",
"1.5.4" => "Programs/Julia 1.5.4/bin/julia.exe",
"1.6.5" => "Programs/Julia-1.6.5/bin/julia.exe",
"1.7.3" => "Programs/Julia-1.7.3/bin/julia.exe",
"1.8.3" => "Programs/Julia-1.8.3/bin/julia.exe",
"1.9.0b" => "Programs/Julia-1.9.0-beta2/bin/julia.exe",
)
jdf = sort(DataFrame(version = collect(keys(julias)), location = collect(values(julias))), :version)
# Check versions
for r ∈ eachrow(jdf)
println("Julia version: ", r.version)
if r.version == "0.6.4"
run(`$(normpath(path, r.location)) -e "println(Pkg.installed()[\"Plots\"])"`);
else
run(`$(normpath(path, r.location)) -e "import Pkg; println(Pkg.installed()[\"Plots\"])"`);
end
println()
end
n = 20
jdf.times = [zeros(n) for _ ∈ 1:nrow(jdf)]
# Run code
for r ∈ eachrow(jdf)
println("Julia version: ", r.version)
pn = normpath(path, r.location)
@showprogress for i ∈ 1:n
if r.version == "0.6.4"
r.times[i] = @elapsed run(`$pn -e "using Plots; scatter(rand(1_000))"`);
else
r.times[i] = @elapsed run(`$pn -e "using Plots; display(scatter(rand(1_000)))"`);
end
end
println()
end
using Plots, Statistics
jdf.average = mean.(jdf.times)
jdf.standard_dev = std.(jdf.times)
bar(jdf.version[2:end], jdf.average[2:end], label = "",
linewidth = 0.1, ylabel = "Time (seconds)", xlabel = "Julia version",
yticks = 0:5:50,
title = "Mean/minimum time for 20 samples\nusing Plots; display(scatter(rand(1_000)))")
scatter!((1:nrow(jdf)-1) .- 0.5, minimum.(jdf.times)[2:end], label = "Minimum", markerstrokewidth = 0.1)
jdf.Plots_version = ["0.17.4", "0.19.3", "1.4.3", "1.4.3", "1.4.4", "1.6.12",
"1.6.12", "1.24.3", "1.27.6", "1.37.2", "1.38.0", "1.38.0-dev"]
select(jdf, :version, :average => (x -> round.(x, digits = 1)) => :average,
:times => (x -> round.(minimum.(x), digits = 1)) => :minimum,
:times => (x -> round.(std.(x), digits = 1)) => :std_dev)