# Slow Julia startup time after sysimage creation (and an unbelievable observation!)

**URL:** <https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102>\
**Category:** Tooling\
**Tags:** repl, startup, sysimage\
**Created:** [August 1, 2020, 8:03pm UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102 "2020-08-01T20:03:04Z")\
**Posts on this page:** 7\
**Page:** 2

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [August 4, 2020, 1:08pm UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/21 "2020-08-04T13:08:03Z")

</div>

Yes! You are right! Then it is much better/closer!

I this case the only thing remained, “let’s tell everyone, if you build it twice you get 4-6x faster startup!” 😃  
Which is GAMECHANGER already! 😃

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [August 4, 2020, 1:41pm UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/22 "2020-08-04T13:41:13Z")

</div>

> [@Marcell\_Havlik](#):
>
> I this case the only thing remained, “let’s tell everyone, if you build it twice you get 4-6x faster startup!” 😃

Well, for someone working on PackageCompiler, understanding why this happens would also be one thing remaining…

It would be useful to figure out if this effect is mostly due to a specific package (perhaps Zygote) and then if there is some specific thing that the package does that causes it.

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [June 23, 2021, 9:49am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/23 "2021-06-23T09:49:56Z")

</div>

Just stumbled upon this interesting thread and decided to see if I can also see the speedup effect when using the double sysimg compile. And I do. Based on @Marcell_Havlik’s script I stripped away some of the larger packages like Zygote and Flux to only leave this (note that this also gets rid of the replacement of the default sysimg and the precompile execution script):

```julia
$ cat sysimage_awesome.jl
import Pkg;
installed_pkgs = Pkg.installed()
function add_pkg(name; kw...)
        !(haskey(installed_pkgs, name) && (!(:version in keys(kw)) || installed_pkgs[name] == kw[:version])) && Pkg.add(Pkg.PackageSpec(;name=name, kw...))
end 
add_pkg("Revise")
add_pkg("BenchmarkTools")
add_pkg("Distributions")
add_pkg("OhMyREPL")
add_pkg("EllipsisNotation")
add_pkg("LanguageServer")
add_pkg("Parameters")
add_pkg("JSON2")
add_pkg("JuliaFormatter")
add_pkg("LightGraphs")
add_pkg("GraphPlot")
add_pkg("Plots")
add_pkg("PackageCompiler")
# SymbolServer
using PackageCompiler
@show "Build starting..."

create_sysimage([:Revise, :JuliaFormatter, 
                :LanguageServer, 
                :PackageCompiler,
                :EllipsisNotation, :Parameters, :JSON2,
                :Distributions,
                :OhMyREPL,
                :Plots,
                ];
        replace_default=false,
        sysimage_path="mysysimage.so")

```

Then compiling the sysimg twice, the second time using the first sysimg:

```julia
$ julia sysimage_awesome.jl 
┌ Warning: Pkg.installed() is deprecated
└ @ Pkg /build/julia/src/julia-1.6.1/usr/share/julia/stdlib/v1.6/Pkg/src/Pkg.jl:566
"Build starting..." = "Build starting..."
[ Info: PackageCompiler: creating system image object file, this might take a while...

$ ls -l *.so
-rwxr-xr-x 1 melis users 315158440 Jun 23 11:39 mysysimage.so
$ mv mysysimage.so mysysimage.first.so

$ julia -J mysysimage.first.so sysimage_awesome.jl 
┌ Warning: Pkg.installed() is deprecated
└ @ Pkg /build/julia/src/julia-1.6.1/usr/share/julia/stdlib/v1.6/Pkg/src/Pkg.jl:566
"Build starting..." = "Build starting..."
[ Info: PackageCompiler: creating system image object file, this might take a while...
$ mv mysysimage.so mysysimage.second.so

$ ls -l *.so
-rwxr-xr-x 1 melis users 315158440 Jun 23 11:39 mysysimage.first.so
-rwxr-xr-x 1 melis users 316299536 Jun 23 11:44 mysysimage.second.so

```

Differences in runtime (for the second time running these commands):

```julia
$ time julia -J mysysimage.first.so -E 'print("Julia $(VERSION) with $(length(Base.loaded_modules)) module");'
Julia 1.6.1 with 137 modulenothing

real	0m0.691s
user	0m0.559s
sys	0m0.130s

$ time julia -J mysysimage.second.so -E 'print("Julia $(VERSION) with $(length(Base.loaded_modules)) module");'
Julia 1.6.1 with 137 modulenothing

real	0m0.306s
user	0m0.192s
sys	0m0.106s

```

Of course, this is a much smaller speedup than OP observes, but it’s still interesting to figure out why this occurs.

Differences in precompile statements:

```julia
$ julia -J mysysimage.first.so -E 'print("Julia $(VERSION) with $(length(Base.loaded_modules)) module");' --trace=stderr
precompile(Tuple{typeof(Base.Threads.resize_nthreads!), Array{URIs.RegexAndMatchData, 1}, URIs.RegexAndMatchData})
precompile(Tuple{typeof(Base.Threads.resize_nthreads!), Array{Base.MPFR.BigFloat, 1}, Base.MPFR.BigFloat})
precompile(Tuple{typeof(Base.Threads.resize_nthreads!), Array{Base.GMP.BigInt, 1}, Base.GMP.BigInt})
precompile(Tuple{ArrayInterface.var"#53#80"})
precompile(Tuple{ArrayInterface.var"#56#83"})
precompile(Tuple{ArrayInterface.var"#110#114"})
precompile(Tuple{ArrayInterface.var"#111#115"})
precompile(Tuple{ArrayInterface.var"#112#116"})
precompile(Tuple{typeof(Base.Libc.Libdl.dlopen), String, UInt32})
precompile(Tuple{typeof(Base.invokelatest), Any})
precompile(Tuple{typeof(Base.Threads.resize_nthreads!), Array{Array{Grisu.Bignums.Bignum, 1}, 1}, Array{Grisu.Bignums.Bignum, 1}})
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, PlotThemes.PlotTheme}, PlotThemes.PlotTheme, Symbol})
precompile(Tuple{typeof(Base.merge), NamedTuple{(), Tuple{}}, Base.Dict{Symbol, Any}})
precompile(Tuple{typeof(Base.foreach), Function, Array{Base.Dict{Symbol, Any}, 1}, Array{Base.Dict{Symbol, Any}, 1}})
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Nothing, Symbol})
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Tuple{Int64, Int64}, Symbol})
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Base.Dict{Any, Any}, Symbol})
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, ColorTypes.RGB{FixedPointNumbers.Normed{UInt8, 8}}, Symbol})
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Float64, Symbol})
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Array{Any, 1}, Symbol})
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Array{Float64, 1}, Symbol})
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Measures.Length{:mm, Float64}, Symbol})
precompile(Tuple{typeof(Base.copyto!), Array{Symbol, 1}, Base.KeySet{Symbol, Base.Dict{Symbol, Any}}})
precompile(Tuple{typeof(Plots.treats_y_as_x), Symbol})
precompile(Tuple{typeof(Base.length), Base.Dict{Base.PkgId, Module}})
precompile(Tuple{typeof(Base.string), String, Base.VersionNumber, Vararg{Any, N} where N})
Julia 1.6.1 with 137 moduleprecompile(Tuple{typeof(Base.show), Nothing})
precompile(Tuple{typeof(Base.show), Base.TTY, Nothing})
nothing
precompile(Tuple{typeof(FileWatching.uvfinalize), FileWatching.FileMonitor})

$ julia -J mysysimage.second.so -E 'print("Julia $(VERSION) with $(length(Base.loaded_modules)) module");' --trace=stderr
precompile(Tuple{ArrayInterface.var"#110#114"})
precompile(Tuple{ArrayInterface.var"#111#115"})
precompile(Tuple{ArrayInterface.var"#112#116"})
precompile(Tuple{typeof(Base.length), Base.Dict{Base.PkgId, Module}})
precompile(Tuple{typeof(Base.string), String, Base.VersionNumber, Vararg{Any, N} where N})
Julia 1.6.1 with 137 moduleprecompile(Tuple{typeof(Base.show), Nothing})
precompile(Tuple{typeof(Base.show), Base.TTY, Nothing})
nothing
precompile(Tuple{typeof(FileWatching.uvfinalize), FileWatching.FileMonitor})

```

```julia
Julia Version 1.6.1
Commit 6aaedecc44* (2021-04-23 05:59 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-4460 CPU @ 3.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.1.0 (ORCJIT, haswell)

```

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [June 23, 2021, 10:10am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/24 "2021-06-23T10:10:13Z")

</div>

I can give you even extreme helps.  
I don’t know when did I did this but I made some advancements and I copy a part of my scripts. I hope it works perfectly for you too!

```julia
import Pkg;
import Base

ENV["JULIA_DEBUG"] = "CUDA" # for better CUDA error messages.

Base.:*(a::Vector{String}, b::Vector{String}) = vcat(a, b) 
Base.:*(a::Vector{Pkg.Types.PackageSpec}, b::Vector{Pkg.Types.PackageSpec}) = vcat(a, b) 

packages = ["Revise"]
packages *= ["BenchmarkTools", "Zygote"]
packages *= ["Flux", "CUDA"]
packages *=["AMDGPU"]
packages *= ["Distributions", "EllipsisNotation", "Parameters", "FastClosures"]
packages *= ["JSON2", "JSON3","CSV", "JLD2", "WebSockets", "HTTP", "FileIO"] # , "JLD", "Redis"
# packages *= ["GraphRecipes"]  
packages *= ["LightGraphs", "GraphPlot", "Colors", "Crayons", "NetworkLayout"]
packages *= ["DataStructures"] # , "ORCA"
packages *= ["PlotlyBase", "Plots"]
packages *= ["KernelDensity", "GeometryTypes", "GeometryBasics", "DataFrames"]
packages *= ["Distributed", "ClusterManagers","Memoize", "Formatting"] # , "LoopVectorization"
packages *= ["AbstractPlotting"]
packages *= ["Makie", "GLMakie"]
packages *= ["PackageCompiler", "PyCall", "JuliaFormatter"] 
packages *= ["Glob", "LRUCache"] # , "KernelDensity", "StatsPlots"
packages *= ["MethodAnalysis", "SnoopCompile", "SnoopCompileCore", "AbstractTrees", "CodeTracking"] # , "ProfileView"
packages *= ["FiniteDifferences"]
packages *= ["DiffRules", "ChainRulesCore", "OrderedCollections", "JuliaInterpreter", "MacroTools"]
Pkg.add(packages)
println("Building packages...")
Pkg.build()

using PackageCompiler
precompileable_pkgs = 
[
  :Revise,  

  # :PrecompilePkg,
  
  :RelevanceStacktrace,

  :PackageCompiler,
  :Distributed,

  :PyCall,

  :Distributions,
  :Flux,
  :Zygote,

  # :CUDA,
  :Colors,
  :Crayons,

  :DataStructures,
  :JSON2,
  :JSON3,
  :JLD2,
  :FileIO,
  :CSV,
  :HTTP,
  :WebSockets,

  :EllipsisNotation,
  :Parameters,
  :Memoize,
  :Glob,

  :MethodAnalysis,
  :SnoopCompile,
  :BenchmarkTools,

  :Formatting,
  :JuliaFormatter,

  # :Plots,
  :GraphPlot, 
  :LightGraphs,
  :NetworkLayout,
  :PlotlyBase,
  :GeometryTypes,
]

syimage_name = "Julia17PreSysimageOpt"
syimage_name_1 = "$(syimage_name)7.so"
syimage_name_2 = "$(syimage_name)8.so" # The second image is double times faster...
println("Building 1. sysimage... ($syimage_name_1)")
create_sysimage(precompileable_pkgs; replace_default = false, sysimage_path="$(syimage_name_1)") # precompile_execution_file = "precompile.jl",
println("Build 1. Done! ($syimage_name_1)")

println("Build 2 started... ($syimage_name_2)")
cmd = """using PackageCompiler; create_sysimage($precompileable_pkgs; replace_default = false, sysimage_path="$(syimage_name_2)")"""
cmd = "'$cmd'"
run(`julia -J$(syimage_name_1) -e $(cmd[2:end-1])`)
println("Build 2 Done! ($syimage_name_2)")

```

This does compile the image with the image you compiled and it gets the speedup. 🙂  
Yeah it would be nice to figure out why do this speedup happen, but still then I enjoy the benefits. 😃

Then I just add the vscode config:

```julia
  "julia.additionalArgs": [
    "-J/home/computername/repo/url/Julia17PreSysimageOpt8.so"
  ],

```

in the end. (Btw I could do it automatically from the script.) Idk why I didn’t automate this yet… maybe for safety 😃

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [June 23, 2021, 11:00am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/25 "2021-06-23T11:00:50Z")

</div>

Some more small tests of the form:

```julia
$ cat genimg.jl
using PackageCompiler

create_sysimage([<packages>]; replace_default=false, sysimage_path="mysysimage.so")

$ julia genimg.jl
$ mv mysysimage.so mysysimage.first.so
$ julia -J mysysimage.first.so genimg.jl
$ mv mysysimage.so mysysimage.second.so
# Differences in runtime (first -> second, run more than once, take lowest)
$ time julia -J mysysimage.first.so -e ''
$ time julia -J mysysimage.second.so -e ''
# Differences in precompile statements 
$ julia -J mysysimage.first.so -e '' --trace=stderr
$ julia -J mysysimage.second.so -e '' --trace=stderr

```

**Revise, JuliaFormatter, LanguageServer, PackageCompiler, OhMyREPL**  
0.248s → 0.174s

Missing with mysysimage.first.so:  
precompile(Tuple{typeof(Base.Threads.resize\_nthreads!), Array{URIs.RegexAndMatchData, 1}, URIs.RegexAndMatchData})  
precompile(Tuple{typeof(Base.Threads.resize\_nthreads!), Array{Base.MPFR.BigFloat, 1}, Base.MPFR.BigFloat})  
precompile(Tuple{typeof(Base.Threads.resize\_nthreads!), Array{Base.GMP.BigInt, 1}, Base.GMP.BigInt})  
precompile(Tuple{typeof(Base.invokelatest), Any})

(some of those `resize_nthreads!` calls don’t really make sense, do they? E.g. `BigFloat` or `RegexAndMatchData`?)

**Revise, PackageCompiler**  
0.217s → 0.122s

Missing with mysysimage.first.so:  
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Base.PkgId, Revise.PkgData}, Revise.PkgData, Base.PkgId})

**Revise**  
0.206s → 0.118s

Missing with mysysimage.first.so:  
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Base.PkgId, Revise.PkgData}, Revise.PkgData, Base.PkgId})

**Plots**  
0.512s → 0.156s

Missing with mysysimage.first.so:  
precompile(Tuple{typeof(Base.Libc.Libdl.dlopen), String, UInt32})  
precompile(Tuple{typeof(Base.invokelatest), Any})  
precompile(Tuple{typeof(Base.Threads.resize\_nthreads!), Array{Base.MPFR.BigFloat, 1}, Base.MPFR.BigFloat})  
precompile(Tuple{typeof(Base.Threads.resize\_nthreads!), Array{Base.GMP.BigInt, 1}, Base.GMP.BigInt})  
precompile(Tuple{typeof(Base.Threads.resize\_nthreads!), Array{Array{Grisu.Bignums.Bignum, 1}, 1}, Array{Grisu.Bignums.Bignum, 1}})  
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, PlotThemes.PlotTheme}, PlotThemes.PlotTheme, Symbol})  
precompile(Tuple{typeof(Base.merge), NamedTuple{(), Tuple{}}, Base.Dict{Symbol, Any}})  
precompile(Tuple{typeof(Base.foreach), Function, Array{Base.Dict{Symbol, Any}, 1}, Array{Base.Dict{Symbol, Any}, 1}})  
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Nothing, Symbol})  
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Tuple{Int64, Int64}, Symbol})  
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Base.Dict{Any, Any}, Symbol})  
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, ColorTypes.RGB{FixedPointNumbers.Normed{UInt8, 8}}, Symbol})  
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Float64, Symbol})  
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Array{Any, 1}, Symbol})  
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Array{Float64, 1}, Symbol})  
precompile(Tuple{typeof(Base.setindex!), Base.Dict{Symbol, Any}, Measures.Length{:mm, Float64}, Symbol})  
precompile(Tuple{typeof(Base.copyto!), Array{Symbol, 1}, Base.KeySet{Symbol, Base.Dict{Symbol, Any}}})  
precompile(Tuple{typeof(Plots.treats\_y\_as\_x), Symbol})

**PackageCompiler**  
Missing with mysysimage.first.so:  
_none!_

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [June 24, 2021, 7:46am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/26 "2021-06-24T07:46:47Z")

</div>

So do I understand well, the second time there are less precompiled directives that goes to stderr?

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [June 24, 2021, 7:53am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/27 "2021-06-24T07:53:19Z")

</div>

Yes, in all cases there are _no_ precompiled directives printed when using the second sysimg

[Previous page](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102.md?page=1)