Using (Web) packages together is much slower, e.g. 0.2 sec + 0.7 sec = 7.8 sec

General Usage
1

I’ll first say, the composability of Julia packages is (otherwise) excellent.

I can e.g. use these two packages together, and these are the times, if loading individually (and exiting in-between):

julia> @time using WebSockets
  0.224824 seconds (135.01 k allocations: 8.546 MiB, 72.83% compilation time)
[restart]
julia> @time using WebIO
  0.693919 seconds (988.58 k allocations: 72.361 MiB, 2.00% gc time, 10.98% compilation time)

but together, as one using (or even two in succession, either order, I’ve before seen order matter):

julia> @time using WebSockets, WebIO
  7.769875 seconds (16.53 M allocations: 692.337 MiB, 6.33% gc time, 90.34% compilation time)

I think I know the reason (invalidations), but hat’s one of the biggest offender I’ve seen. I would like to know a good solution. I was looking into SnoopCompile[Core] and some other tricks I know of. If someone can fix (even only?) these two packages, then the Julia community will be grateful.

These are the dependencies of Blink.jl that I thought was being slow, but it’s not to blame nor any individual dependency (all relatively fast individually):

@time using Base64, BinDeps, Distributed, JSExpr, JSON, Lazy, Logging, MacroTools, Mustache, Mux, Reexport, Sockets, WebIO, WebSockets

I was looking into why the seemingly awesome (very recently registered package) CMPlot.jl is slow to start (13 sec. seemed excessive), because of PlotlyJS (8.5 sec) mostly it seemed, but actually it’s mostly “because” of Blink (8.1 sec).

One solution is:

$ ~/julia-1.7.0-rc3/bin/julia -O0 --compile=min

julia> @time using Blink
  1.384684 seconds (2.06 M allocations: 131.663 MiB, 5.32% gc time, 6.69% compilation time)

and similar for only:

julia> @time using CMPlot
  4.209774 seconds (6.11 M allocations: 392.835 MiB, 8.00% gc time, 2.60% compilation time)

Note, this isn’t about web programming [packages] per se. people may not realize they’re using web pages when “just” plotting (Blink brings in a full Chrome web browser, it’s just embedded local web browser and server).

This may all be bloated for disk space (I don’t really care), and people may blame the speed unnecessarily on that.

1 Like
2

I got “loading” these two packages from 7.7 sec. to 0.8 sec, by commenting out three lines, that I thought might break WebIO, but it doesn’t seem to.

Could the problem be these three lines, I disabled:

Requires is a Julia package that will magically make loading packages faster, maybe.

That’s a big “maybe”, anyone know why that is stated? Note, Blink has both packages as dependenices, and WebIO has WebSockets as a dependency and the example code for Blink still works. At runtime it was always slow at first (back when I last used it), and still is, maybe I just shifted the waiting from using to actually using the package.

Loading CMPlot is down to 6.1 sec. on defaults, and 3.8 sec on non-defaults.

3

WebIO loads a bunch of extra code conditionally when WebSockets is loaded. So it isn’t surprising that the load time is longer when they are loaded together in the same session.

1 Like
4

Another example (MWE) and regression from 1.6:

julia> @time using TimeZones, CategoricalArrays
  2.458267 seconds (3.90 M allocations: 219.764 MiB, 2.19% gc time, 86.95% compilation time)

both are under 0.3 sec separately, so this seems strange. The packages seem like they should be completely orthogonal. I was originally debugging why RData has slow loading i.e. lead to:

using DataFrames, CategoricalArrays, FileIO, TimeZones, Unicode

This even happens if I only import parts, e.g. something random:

julia> @time import TimeZones: hour
  0.232950 seconds (266.72 k allocations: 22.637 MiB, 21.89% compilation time)

julia> @time import CategoricalArrays: recode
  2.290661 seconds (3.63 M allocations: 197.128 MiB, 2.71% gc time, 92.66% compilation time)

Then a follow-up question, if I import only some things, can and should it be quicker than importing/using the whole package? It doesn’t seems to in general, or never, but likely could?

This is a regression from 1.6.0:

julia> @time import TimeZones: hour
  0.210675 seconds (186.50 k allocations: 18.328 MiB, 3.59% compilation time)

julia> @time import CategoricalArrays: recode
  0.671600 seconds (939.14 k allocations: 57.319 MiB)

while not faster than:

julia> @time import TimeZones, CategoricalArrays
  0.832828 seconds (1.13 M allocations: 75.642 MiB, 1.25% gc time, 0.82% compilation time)
5

It is as expensive as importing the whole package. There is no smaller granularity to import thing than package level. I had a proposal for reducing that to submodules. Proposal for allowing packages to opt-into `import A.B` only loading `B` without loading `A`. · Issue #2005 · JuliaLang/Pkg.jl · GitHub.

1 Like