Package load speed: Windows vs Linux

tfiers
Mon, 16 Jan 2023 19:04:48 +0000
Performance

I have a feeling that Julia package import times are significantly faster on Linux than on Windows.
See e.g. this thread where:

@giordano’s seven-year-old Linux laptop loads Unitful in 0.44 seconds

my two-year-old Windows laptop (i7, 8-core, SSD) loads Unitful in 1.38 seconds (3x as slow)

(Both measurements were made in a fresh tmp env, in a --startup-file=no session. My Julia version was 1.8.1)

Quoting Mosè Giordano in that same thread, “I/O is pretty terrible on Windows”

(For an explanation of why Windows I/O is slower than Linux, see e.g. these two reddit threads: one, two. The second links to "I Contribute to the Windows Kernel. We Are Slower Than Other Operating Systems. Here Is Why.")

With native code caching in upcoming Julia versions, these import times will increase further (see left column in image below)

So, I’m a bit worried. (Especially because, I presume, most core devs use Linux. And thus, loading more small files might feel like a relatively cheap operation – correct me if I’m wrong).

Should I switch to Linux? I’ll miss PowerPoint.
Should Julia package loading do something different on Windows? (A shot in the dark: maybe bundling many small cache files together in one big tarball would help, to pay the I/O tax only once, instead of many times)

Nice plot by @msjgriffiths (source) of Tim Holy’s data in #47184:

25 posts - 11 participants