First Pluto notebook launches are slower on Julia 1.9 beta 3

fonsp
Mon, 23 Jan 2023 20:46:06 +0000
Internals & Design

Hi! Today, I ran some benchmarks to compare real-life launch times for first-time users of Julia 1.9.0-beta3 vs Julia 1.8.5. That means people who just installed Julia on their computer, and they are excited to get started!

Results

Summary of three samples, details below.

I found that starting Pluto the first time is 2.3x slower in Julia 1.9 (57.8 sec vs 25.4 sec). This is common in Julia 1.9: the increase in precompilation time is greater than the reduction in TTFX.

Once Pluto is running, running this basic data science notebook (Plots, CSV) is 1.6x slower (3 min 16 sec vs 2 min 2 sec).

This was on a very fast computer, these times might easily be double on an old laptop. More details about my benchmark here.

Bad first impression

In my experience (talking with university teachers and Pluto users), slow launch times are a big pain point of using Pluto and Julia. Teachers report that they worry about leaving a bad first impression.

It is easy to forget that launch times are experienced as disproportionally large by students because:

They have an empty .julia cache.

They use lower-power computers than their teachers.

They run many different notebooks, each with a new Manifest to be loaded (which might not have much overlap with previous notebooks). They spend more time loading, less time tweaking.

They are used to faster digital experiences (spreadsheets, email). Normally, seeing a loading bar stuck for 1 minute means that something is broken.

They might be used to faster programming environments (javascript, python). For example, this notebook, written in JavaScript, also loads a CSV dataset, runs some basic statistics and displays plots. With an empty cache, it downloads and runs in your browser in 3 seconds. That’s 100x faster!

Outside of education, I only have limited experience with first-time Julia users. But here, my worry is that during those extra minutes of loading bars, we are scaring away new users, without a teacher to tell them that this is expected (and better times are ahead).

My conclusions

On one hand, as an experienced Julia user, I really appreciate the improved TTFX when working in a constant package environment. And I am super impressed by the technical achievement of the feature!

But as someone trying to make Julia look fast and interactive on the web, and accommodating for first-time Julia users, I wanted to write this post to highlight this perspective. In its current state, Julia 1.9 is much slower for first-time users. Are we making a trade-off at the expense of new users?

The way I see it: On their first day, students would enjoy Julia 1.8 more than Julia 1.9, they are more likely to pick it up again, and to recommend Julia to their friends.

Solutions

The feature also came with a new command-line flag, julia --pkgimages=(yes|no). This empowers us to decide when to use or not use the feature, and we could potentially get the best of both worlds!

For Pluto.jl, we are currently discussing setting this to no by default for notebook processes and precompilation. Or we might find a dynamic approach, e.g. only using pkgimages when you run a notebook a second time, or using new GUI to let users decide.

Outside of Pluto.jl, what about setting this flag to no by default in Julia? This way, new users have a better first impression, and more experienced Julia users can use julia --pkgimages=yes to benefit from the feature. What do others think about this?

158 posts - 54 participants