Any way to load a module faster even in a fresh session (given the module has been loaded in a previous sessoin)?

General Usage Performance
1

I know that first time a module is loaded in a new session, Julia needs to go through a lot of compiling so it’s slow, and successive loads are much faster. But I read that recent versions of Julia will pre-compile modules and cache them automatically. I believe this means next time loading a module, even for the first time in a new session, should be fast as per-compiled cache will be used, as long as there are no changes to the module and its dependencies.

However, in my experience of JuliaPro 1.01 I noticed only the modules that came with JuliaPro installation (e.g. LinearAlgebra) are super fast to load; anything else still takes quite some time, especially the plotting packages. Since I use quite a few modules including PyPlot in a typical script, it would take well over a minute to load modules in a fresh Julia session.

I know successive runs will be very fast as long as I don’t close the terminal, but when I try to promote Julia to Python and Matlab users, the demo would often be in a fresh terminal; their own test runs would often be in a fresh terminal; when I need to do quick calculations during a discussion with my colleagues, it’ll be in a fresh terminal. It’s embarrassing to have to show an empty terminal for tens of seconds while CPU usage and laptop fan crank up high, after I had claimed to them Julia is super fast.

Are the commonly used big modules really pre-compiled or did I do something wrong? Is there a way to make loading modules fast, even in a fresh session, as long as it has been loaded on the same computer once, and there is no change to it and its dependencies?

Below shows how much time several typical modules take to load in a fresh Julia session on my work laptop (a lousy Intel Core i7-6600U CPU with 8GB RAM, Windows 10 64-bit).

using Revise: 1.385904 seconds (487.14 k allocations: 27.165 MiB, 0.95% gc time)
using LinearAlgebra: 0.963156 seconds (944.32 k allocations: 44.840 MiB, 1.76% gc time)
using Plots: 13.692310 seconds (17.96 M allocations: 949.593 MiB, 7.85% gc time)
gr(): 1.209818 seconds (313.23 k allocations: 15.442 MiB, 3.07% gc time)
using DataFrames: 18.728653 seconds (26.70 M allocations: 1.330 GiB, 5.27% gc time)
using CSV: 21.418905 seconds (25.34 M allocations: 1.193 GiB, 6.11% gc time)
using PyPlot: 41.934533 seconds (41.05 M allocations: 2.514 GiB, 2.79% gc time)

Thanks!!

2

When packages precompile, they get saved in a lowered form, but not as a binary (like the system image). That means LLVM still has to compile them. But LLVM should be built into the system image already.

PackageCompiler lets you build other libraries into the system image. I haven’t tested it much, but it should at least work with the plotting library Makie.

You could look into it.

I believe there will be many improvements long term.

1 Like
3

Thanks! I wanted to try PackageCompiler, but JuliaPro registry has not included it yet:

(JuliaPro_v1.0.1.1) pkg> add PackageCompiler
  Updating registry at `C:\JuliaPro-1.0.1.1\pkgs-1.0.1.1\registries\JuliaPro`
  Updating git-repo `https://pkg.juliacomputing.com/registry/JuliaPro`
ERROR: The following package names could not be resolved:
 * PackageCompiler (not found in project, manifest or registry)
Please specify by known `name=uuid`.

I saw that the General registry included it:

However, I don’t know how to switch to use the General registry. I did find string “https://pkg.juliacomputing.com/” inside C:\JuliaPro-1.0.1.1\Julia-1.0.1\etc\julia\startup.jl and tried replacing it with “https://github.com/JuliaRegistries/General/” then restarting Julia, but that didn’t seem to work.

I also tried directly adding from GitHub, but was soon led to a rabbit hole of dependency tree:

(JuliaPro_v1.0.1.1) pkg> add https://github.com/JuliaLang/PackageCompiler.jl
  Updating git-repo `https://github.com/JuliaLang/PackageCompiler.jl`
[ Info: Assigning UUID e7f35708-9c4f-53ac-b576-fe1e8b0e8cf1 to PackageCompiler
 Resolving package versions...
ERROR: The following package names could not be resolved:
 * ArgParse (not found in project, manifest or registry)
 * SnoopCompile (not found in project, manifest or registry)
Please specify by known `name=uuid`.

etc. etc.

Do you know how to switch JuliaPro to use the General registry (temporarily)? Thanks!