# Slow Julia startup time after sysimage creation (and an unbelievable observation!)

**URL:** <https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102>\
**Category:** Tooling\
**Tags:** repl, startup, sysimage\
**Created:** [August 1, 2020, 8:03pm UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102 "2020-08-01T20:03:04Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [August 1, 2020, 8:03pm UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/1 "2020-08-01T20:03:04Z")

</div>

Hey Julianers,

I found this:

```julia
➜ julia-awesomeness git:(master) ✗ time julia -e ""         
julia -e "" 0,10s user 0,09s system 181% cpu 0,110 total

```

I built a [sysimage](https://github.com/Cvikli/julia-awesomeness/blob/master/sysimage_awesome.jl) and got these numbers:

```julia
➜ julia-awesomeness git:(master) ✗ time julia -e ""                      
julia -e "" 4,84s user 0,67s system 117% cpu 4,684 total

```

Can someone help me how to build a sysimage that has the same startup time like the original “julia”?

I think the solution to this could be a really big deal for anyone who use julia or coming from python.

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [August 2, 2020, 4:27am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/2 "2020-08-02T04:27:39Z")

</div>

> [@Marcell\_Havlik](#):
>
> Can someone help me how to build a sysimage that has the same startup time like the original “julia”?

**imho: probably more modules ( 152 vs. 33 ) … more time ??**

I don’t know

- your julia environment ( 1.4? 1.5? )
  - the 1.5 is better

- your number of Base.loaded\_modules

so I tried to replicate your times … but your script is not worked for me …

So with your prefered packages - without package versions ( see [my dockerfile](https://gist.github.com/ImreSamu/235daa6426076d5167a0774ea96c0cb0) ) …  
I see **152 modules**

```julia
root@88ce0d0df7d7:/# time julia -E "println(length(Base.loaded_modules));VERSION"
152
v"1.5.0-rc2.0"

real	0m3.464s
user	0m3.504s
sys	0m0.938s

```

the default 33 modules time tested on `docker run -it --rm julia:1.5 bash`

```julia
root@5bf6a36abe91:/# time julia -E "println(length(Base.loaded_modules));VERSION"
33
v"1.5.0-rc2.0"

real	0m0.528s
user	0m0.783s
sys	0m0.535s

```

the reference Julia 1.4 results is more ( ~4.363s ) … so the v1.5 version is better.

```julia
root@b2262a156039:/# time julia -E "println(length(Base.loaded_modules));VERSION"
152
v"1.4.2"

real	0m4.363s
user	0m4.346s
sys	0m0.936s

```

> [@Marcell\_Havlik](#):
>
> I think the solution to this could be a really big deal for anyone who use julia or coming from python.

As I know the latency is high priority … you can find more info in this video

[![](https://global.discourse-cdn.com/julialang/original/3X/2/5/25d92fc8d496bba03c1793e16cb4c78a684e8de5.jpeg "JuliaCon 2020 | Shippable apps and custom sysimages with PackageCompiler.jl | Kristoffer Carlsson") ](https://www.youtube.com/watch?v=d7avhSuK2NA&t=387)

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [August 2, 2020, 8:30am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/3 "2020-08-02T08:30:17Z")

</div>

Thank you for the test command.  
My version was 1.5.0-rc2. The test:

```julia
➜ ~ time julia -E 'print("Julia $(VERSION) with $(length(Base.loaded_modules)) module");""' 
Julia 1.5.0-rc2.0 with 152 module""
julia -E 7,89s user 0,72s system 110% cpu 7,795 total

```

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [August 2, 2020, 8:56am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/4 "2020-08-02T08:56:01Z")

</div>

I rerun the whole create\_sysimage and now my timings are:

```julia
➜ ~ time julia -E 'print("Julia $(VERSION) with $(length(Base.loaded_modules)) module");""'  
Julia 1.5.0-rc2.0 with 152 module""
julia -E 1,53s user 0,66s system 161% cpu 1,357 total

```

This is crazy! 😮  
What is going on?

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [August 2, 2020, 11:03am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/5 "2020-08-02T11:03:40Z")

</div>

One of my friend also tried with other versions of julia (1.4.2 and 1.5.0-rc1) and this works! We have to build the sysimage double times and then it dreceases the startuptime with 5-8x!!! 😮

Any master has any idea on this?

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [August 3, 2020, 3:48am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/6 "2020-08-03T03:48:25Z")

</div>

> [@Marcell\_Havlik](#):
>
> We have to build the sysimage double times and then it dreceases the startuptime with 5-8x!!!

ouch 🤔, I can’t replicate in my docker environments.  
I have a near same results ( ~ same time )

Would you please re-create your method in a Dockerfile ?

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [August 3, 2020, 7:41am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/7 "2020-08-03T07:41:30Z")

</div>

Can you send me an example Dockerfile and the steps you use. So I can modify the workflow to reproduce this case.

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [August 3, 2020, 10:48am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/8 "2020-08-03T10:48:59Z")

</div>

- [Julia1.5rc2 Package compiler test 2020Aug03 · GitHub](https://gist.github.com/ae5e35434567f713f7537508fa5f4dbe)
  - `Dockerfiles.*` + `test.sh`

### jpct15base simple package recompile

```julia-auto
33
v"1.5.0-rc2.0"

real	0m0.474s
user	0m0.820s
sys	0m0.685s

```

### jpct15t1 trace + create\_sysimage

```julia-auto
152
v"1.5.0-rc2.0"

real	0m3.523s
user	0m3.618s
sys	0m0.942s

```

### jpct15t2 trace + create\_sysimage(traced.jl) + trace2 + create\_sysimage2(traced2.jl)

```julia-auto
152
v"1.5.0-rc2.0"

real	0m3.479s
user	0m3.560s
sys	0m0.952s

```

### jpct15c2 trace + 2x create\_sysimage(traced.jl)

```julia-auto
152
v"1.5.0-rc2.0"

real	0m3.485s
user	0m3.566s
sys	0m0.882s

```

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [August 3, 2020, 11:02am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/9 "2020-08-03T11:02:18Z")

</div>

Is there a difference when starting julia with `--trace-compile=stderr` when using the two different sysimages. I’m thinking maybe the first one has some unresolved invalidations in it that causes recompilation. But that’s just a guess.

Also, if possible could you test on julia master.

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [August 3, 2020, 11:31am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/10 "2020-08-03T11:31:26Z")

</div>

> [@kristoffer.carlsson](#):
>
> Also, if possible could you test on julia master.

it is not easy …  
as I see some packages ( `Plot` ) not `1.6.0-DEV.572 (2020-08-02)` compatible yet …  
so we need to find - compatible packages …

```julia
@v1.6) pkg> precompile
Precompiling project...
[Info: Precompiling Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]
ERROR: LoadError: InitError: could not load library "/root/.julia/artifacts/08ab4b2de494e1d61e581ab1fbd610b0b4d1876f/lib/libavdevice.so"
libx264.so.157: cannot open shared object file: No such file or directory
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl ./libdl.jl:114

```

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [August 3, 2020, 11:33am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/11 "2020-08-03T11:33:49Z")

</div>

Ok, I had to realise I am not good at docker managment. I had an error and didn’t have time to solve it.  
`ERROR: LoadError: InitError: could not load library "/root/.julia/artifacts/08ab4b2de494e1d61e581ab1fbd610b0b4d1876f/lib/libavdevice.so"`

I think you did it right.  
I tested it on 2 computer with different setups and I had to double build to get this extreme speedup. Essentially I am just running this file twice:  
[https://github.com/Cvikli/julia-awesomeness/blob/master/sysimage\_awesome.jl](https://github.com/Cvikli/julia-awesomeness/blob/master/sysimage_awesome.jl)

I hope you can achieve the same!

I can say for the first build I had 4.5-5.0 second startuptime and after the second build I have 0.9s which is totally acceptable.

I will try to build the master, but I have to work now. Maybe tomorrow!

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [August 3, 2020, 11:43am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/12 "2020-08-03T11:43:49Z")

</div>

> [@kristoffer.carlsson](#):
>
> Is there a difference when starting julia with `--trace-compile=stderr` when using the two different sysimages.

thank you for the tips.

I have added the `--trace-compile=stderr` to my test script …  
the result:

- [Julia1.5rc2 Package compiler test 2020Aug03 · GitHub](https://gist.github.com/ImreSamu/b286d0f3e7f031169af1e9097c91ca3a#file-output-md)

interesting … lot of `CUDA` related precompile …

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [August 3, 2020, 7:41pm UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/13 "2020-08-03T19:41:44Z")

</div>

> [@Marcell\_Havlik](#):
>
> I rerun the whole create\_sysimage and now my timings are:

Do you also replace the default sysimage when running it the first time? So the second time you run PackageCompiler, it runs with the new sysimage?

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [August 3, 2020, 11:45pm UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/14 "2020-08-03T23:45:56Z")

</div>

I have updated my test script - see [output.md](https://gist.github.com/ImreSamu/4b3168b83bfffd1d337619a81f2dddbc#file-output-md)

- [test.sh](https://gist.github.com/ImreSamu/4b3168b83bfffd1d337619a81f2dddbc#file-test-sh) + Dockerfiles

And now testing this command

```julia
   docker run -it --rm $1 bash -c 'time julia \
              --trace-compile=stderr \
              -E "using \
                    Revise, JuliaFormatter, \
                    LanguageServer, \
                    PackageCompiler, \
                    EllipsisNotation, Parameters, JSON2, \
                    Distributions, \
                    OhMyREPL, \
                    Plots, \
                    Flux, Zygote; \
                   println(length(Base.loaded_modules));VERSION" \
              ' >> output.md

```

`real	2m20.329s` only precompile :  
`real	0m3.822s` precompile + trace + create\_sysimage  
`real	0m3.610s` precompile + 2x( trace + create\_sysimage ) **(a little faster!)**  
`real	0m3.711s` precompile + trace + 2x( create\_sysimage )

2nd. run : ( `cat output.md | grep real` )

| real | time |
| --- | --- |
| real | 2m22.098s |
| real | 0m3.637s |
| real | 0m3.554s |
| real | 0m3.558s |

3rd. run :

| real | time |
| --- | --- |
| real | 2m26.261s |
| real | 0m3.753s |
| real | 0m3.695s |
| real | 0m3.662s |

As I see … a little faster after the second sysimage creation ( 3. and 4. row)

On the other hand …  
**original: 140sec vs. sysimage: 4sec  
imho ~35x faster in this extreme case !!!**

my test environment:

- Thinkpad T480s i5; ubuntu20.04 ; Turbo boost OFF → 1.6Ghz )

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [August 4, 2020, 7:10am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/15 "2020-08-04T07:10:32Z")

</div>

Yes-yes. That could be the key. But you can run the same file I run:  
[https://github.com/Cvikli/julia-awesomeness/blob/master/sysimage\_awesome.jl](https://github.com/Cvikli/julia-awesomeness/blob/master/sysimage_awesome.jl)

And remember you can always restore the state! :

```julia
using PackageCompiler
restore_default_sysimage()

```

My script was tried by some of my friends so I can tell you it is independent from the julia version and the hardware.

I really suggest everyone to run that script two times in a row and just check it out, it will be damn fast.

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [August 4, 2020, 7:13am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/16 "2020-08-04T07:13:21Z")

</div>

@ImreSamu did you try the script I linked and it didn’t bring the speedup?

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [August 4, 2020, 9:25am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/17 "2020-08-04T09:25:59Z")

</div>

> [@Marcell\_Havlik](#):
>
> did you try the script I linked and it didn’t bring the speedup?

as I see now the `precompile.jl` is exist

- but empty … is it correct?

I have added 4 extra test cases … all Dockerfile ( 8 ) + test.sh uploaded

- [Julia1.5 Package compiler test 2020Aug04 with sysimage\_awesome · GitHub](https://gist.github.com/f88ca40d189d47ae1bf773d33ecdcd60)

my result:

reference  
real 1m59.897s ( only with precompile )  
real 0m3.510s ( trace + sysimage )  
real 0m3.469s  
real 0m3.442s

[with sysimage\_awesome.jl](https://github.com/Cvikli/julia-awesomeness/blob/master/sysimage_awesome.jl)  
real 0m10.778s ( run 1x sysimage\_awesome )  
real 0m3.481s ( run 2x sysimage\_awesome )  
real 0m3.444s ( run 3x sysimage\_awesome )  
real 0m3.452s ( run 4x sysimage\_awesome )

the sysimage\_awesome first start time is high (10.7s ) !  
and sysimage\_awesome second run is similar with my original start times. ( ~ 3.5s)

strange … but I don’t know the “Why?”

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [August 4, 2020, 9:38am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/18 "2020-08-04T09:38:32Z")

</div>

> [@Marcell\_Havlik](#):
>
> I really suggest everyone to run that script two times in a row and just check it out, it will be damn fast.

in the official doc : [https://julialang.github.io/PackageCompiler.jl/dev/examples/ohmyrepl/](https://julialang.github.io/PackageCompiler.jl/dev/examples/ohmyrepl/)  
there is a `--trace-compile` part …

> The first goal is to have Julia emit the functions it compiles when running OhMyREPL. To this end, start Julia with the `--trace-compile=ohmyrepl_precompile.jl` flag.

but this part is missing from your script …

- no `--trace-compile=precompile.jl`
- and the `precompile.jl` is empty

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [August 4, 2020, 11:58am UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/19 "2020-08-04T11:58:46Z")

</div>

Thank you for pointing out!

Thank you for the tests, as I see, there is a 10.7s and a 3.48s about 3x speed increase, which can actually mean it is working in your case!  
I measure my speedups on a:

```julia
AMD® Ryzen threadripper 1950x 16-core processor × 32
128GB DDR4 4 channel
3200MBit/s 1TB SSD
Ubuntu 20.04.

```

Your speedup from 10.8s to 3.5s  
Mine is from 7.8s to 1.3s which is like a 1.5 times more speedup and maybe could be associated with the hardware diff, I guess.

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [August 4, 2020, 12:57pm UTC](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102/20 "2020-08-04T12:57:55Z")

</div>

> [@Marcell\_Havlik](#):
>
> Mine is from 7.8s to 1.3s which is like a 1.5 times more speedup and maybe could be associated with the hardware diff, I guess.

IF I test the simple : `'time julia -e "" '` command ( as you suggested in the first post )  
THEN the real times:  
real - with tracing  
real 0m0.239s ( only with precompile )  
real 0m2.476s ( trace + sysimage )  
real 0m2.438s  
real 0m2.413s  
real - sysimage\_awesome  
real 0m9.644s ( run 1x sysimage\_awesome )  
real 0m2.457s ( run 2x sysimage\_awesome )  
real 0m2.427s ( run 3x sysimage\_awesome )  
real 0m2.430s ( run 4x sysimage\_awesome )

so 9.6s/2.4s = **4x** ( and yours 7.8/1.3 = **6x** )

[Next page](https://discourse.julialang.org/t/slow-julia-startup-time-after-sysimage-creation-and-an-unbelievable-observation/44102.md?page=2)