# Taking TTFX seriously: Can we make common packages faster to load and use

**URL:** <https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949>\
**Category:** Performance\
**Tags:** ttfp\
**Created:** [January 20, 2022, 5:16pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949 "2022-01-20T17:16:03Z")\
**Posts on this page:** 20\
**Page:** 5

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [January 27, 2022, 3:13pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/88 "2022-01-27T15:13:45Z")

</div>

I’m talking about the reality of how it works now. By “breaks” precompilation I mean precompiling unstable functions seems to give less or no reduction in TTFX compared to precompiling stable functions.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [January 27, 2022, 3:29pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/89 "2022-01-27T15:29:34Z")

</div>

The basic reason for this is that with type stable code, the compiler can trace all the functions your top level code depends on and precompile all of them, but if the compiler has to trace unstable functions, it loses the ability to know what methodInstances get called which means it can’t precompile the recursive dependencies.

---

<div class="post-metadata">

**Author:** ![antoine-levitt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antoine-levitt/32/4008_2.png) [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)\
**Post date:** [January 27, 2022, 3:37pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/90 "2022-01-27T15:37:09Z")

</div>

> The basic reason for this is that with type stable code, the compiler can trace all the functions your top level code depends on and precompile all of them, but if the compiler has to trace unstable functions, it loses the ability to know what methodInstances get called which means it can’t precompile the recursive dependencies.

I get that it applies to `precompile` statements, but does it also apply to function calls?

---

<div class="post-metadata">

**Author:** ![mihalybaci](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mihalybaci/32/13528_2.png) [@mihalybaci](https://discourse.julialang.org/u/mihalybaci)\
**Post date:** [January 27, 2022, 3:46pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/91 "2022-01-27T15:46:12Z")

</div>

My only experience in developing packages is a little toy one I created just to learn more about the process, but I have not yet found the need to develop my own “serious” package yet. So, in a sense, I feel like this thread is directly towards people like myself who will need a good deal of help to avoid common mistakes/programming patterns that increase the TTFX. Rightly or wrongly, reading the comments gives the impression that there is a very high bar to clear.

Others ([here](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/84) and [here](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/29)) have come up with good “precompilation checklists” to go through for reducing start time. But now consider a “knowledge checklist” one may need in order to implement those suggestions (and the others above):

- how to interpret the built-in profiler,
- how to recognize type instabilities and understand `@code_warntype`,
- how to read a ProfileViews.jl flamegraph,
- how to use Cthulhu.jl,
- what causes an “invalidation”,
- how to use SnoopCompile.jl,
- how to “properly” use Requires.jl
- what makes code `precompile`-able in the first place,

and there are likely others I’ve missed. Some of these are fairly basic (interpreting the profiler and `@code_warntype`, reading flame graphs), but others may require more understanding.

Looking at the entirety of the thread, what would help the most is an easy-to-follow tutorial, perhaps with a dummy package repo on github, where each step is shown and explained in order of “low-hanging fruit” to “the hard stuff”. Because right now, again rightly or wrongly, I look at this thread and just think “wow, that’s a lot of work”, so anything to make the process seem more accessible would be huge.

---

<div class="post-metadata">

**Author:** ![ranocha](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ranocha/32/35588_2.png) [@ranocha](https://discourse.julialang.org/u/ranocha)\
**Post date:** [January 27, 2022, 6:54pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/92 "2022-01-27T18:54:25Z")

</div>

> [@Palli](#):
>
> Do you mean the precompilation time only, or does it also affect `using` (since precompilation isn’t full, some of the compilation is deferred)?

I mean the first time a function is called.

---

<div class="post-metadata">

**Author:** ![lostella](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lostella/32/356_2.png) [@lostella](https://discourse.julialang.org/u/lostella)\
**Post date:** [January 28, 2022, 1:33pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/93 "2022-01-28T13:33:10Z")

</div>

Maybe this is stupid but: I noticed that [`@inferred`](https://docs.julialang.org/en/v1/stdlib/Test/#Test.@inferred) was never mentioned in the thread. I know that other tools allow for more fine-grained inspection, but shouldn’t `@inferred` allow to catch at least a few type instabilities?

(I’m asking since I use it all the time)

---

<div class="post-metadata">

**Author:** ![mihalybaci](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mihalybaci/32/13528_2.png) [@mihalybaci](https://discourse.julialang.org/u/mihalybaci)\
**Post date:** [February 1, 2022, 7:54pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/94 "2022-02-01T19:54:33Z")

</div>

Inspired by [this post](https://discourse.julialang.org/t/ttfx-with-dataframes-and-csv/75432/34) on TTFX with CSV/DataFrames, I made a quick attempt at a function to run script files repeatedly.

```julia
using Statistics

"""
    ttfx(; code="sleep(1)", N=10, args = "", preview=false)

Compute the time to first X. 

`ttfx` will run a `file` `N` times to determine the total 
startup cost of running certain packages/functions. `ttfx()`
with no arguments will simply run `sleep(1)` and may be used
to estimate the base julia runtime cost.

`code` can either be a short snippet that will run with 
the `-e` switch or a file containing the script to be run.

`args` can be used to set Julia runtime options

`preview = true` will show the final command without running it.

"""
function ttfx(; code="sleep(1)", N=10, args = "", preview=false)
    # If running a short snippet and not a file, add a -e
    if !isfile(code)
        code = "-e '$code'"
    end

    # `cmd doesn't interpolate properly or something
    # so using shell_parse`and cmd_gen is the workaround
    ex, = Base.shell_parse("julia $args $code")
    julia_cmd = Base.cmd_gen(eval(ex))

    # Return only the command that would have been run
    preview && return julia_cmd
    
    # Run the command N times
    times = Vector{Float64}(undef, N)
    for i = 1:N
        times[i] = @elapsed run(`$julia_cmd`)
    end

    return median(times), times
end

# Run the default timing with sleep
t = ttfx()

# Run `using CSV` 15 times in the current project with CSV.jl installed and 8 threads
t = ttfx(code="using CSV", N=15, args="-t 8 --project=@.")

```

For me `ttffx()` takes a median time of ~1.17 seconds, so there is a baseline julia runtime cost of 0.17 second on my machine. Then `using CSV` on my computer takes a median time of 3.4 seconds

Feel free to edit and expand this (perhaps into the `@ctime` macro that was suggested). Code suggestions welcome!

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [February 2, 2022, 1:34am UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/95 "2022-02-02T01:34:11Z")

</div>

You can clean up the `Cmd` stuff by doing it like this:

```julia
function ttfx(; code="sleep(1)", N=10, args = String[], preview=false)
    # If running a short snippet and not a file, add a -e
    if !isfile(code)
        code = `-e $code`
    end

    julia_cmd = `julia $args $code`

    # Return only the command that would have been run
    preview && return julia_cmd
    
    # Run the command N times
    times = Vector{Float64}(undef, N)
    for i = 1:N
        times[i] = @elapsed run(julia_cmd)
    end

    return median(times), times
end

```

In this way, to pass `args`, do pass a list of strings like `ttfx(; args=["-O1","--compile=min"])`.

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [February 2, 2022, 9:17am UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/96 "2022-02-02T09:17:19Z")

</div>

> [@ericphanson](#):
>
> In this way, to pass `args` , do pass a list of strings like `ttfx(; args=["-O1","--compile=min"])` .

Or `ttfx(args=`-O1 --compile=min`)` works too.

---

<div class="post-metadata">

**Author:** ![mihalybaci](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mihalybaci/32/13528_2.png) [@mihalybaci](https://discourse.julialang.org/u/mihalybaci)\
**Post date:** [February 2, 2022, 2:50pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/97 "2022-02-02T14:50:01Z")

</div>

Thanks! I initially tried using both the backticks `` and `@cmd` to convert the string to `Cmd`, but I was getting a strange error from `Base.shell_parse`, which is the reason for that note in my original function. But for some reason your new version doesn’t give me the same error…

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [February 2, 2022, 3:35pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/98 "2022-02-02T15:35:23Z")

</div>

Strings, commands, and arrays all interpolate into commands differently, and when you tried, you probably you had something as a string that should’ve been a command or something like that. I often end up using trial-and-error although I really should just learn the rules. In the end, you should usually not need “manual quoting” like `code = "-e '$code'"`, and definitely should not need `eval` or `Base.shell_parse`, etc.

---

<div class="post-metadata">

**Author:** ![mihalybaci](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mihalybaci/32/13528_2.png) [@mihalybaci](https://discourse.julialang.org/u/mihalybaci)\
**Post date:** [February 2, 2022, 5:12pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/99 "2022-02-02T17:12:09Z")

</div>

Ah, that’s good to know. Trial-and-error is how I ended up with what I had initially.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [February 3, 2022, 7:36pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/100 "2022-02-03T19:36:38Z")

</div>

Heres a minor rework, with a macro:

```julia
using Statistics
function ttfx(; code="sleep(1)", N=2, args = String[], preview=false)
    jl_startup = @elapsed run(`julia $args -e ""`)
    # If running a short snippet and not a file, add a -e
    if !isfile(code)
        code = `-e $code`
    end
    julia_cmd = `julia $args $code`
    # Return only the command that would have been run
    preview && return julia_cmd
    # Run the command N times
    times = Vector{Float64}(undef, N)
    for i = 1:N
        times[i] = @elapsed run(julia_cmd)
    end
    times .-= jl_startup
    return (mean=mean(times), times=times)
end
macro ttfx(code::String, N=1)
    ttfx(; code, N)
end

@ttfx "your code here" 10

```

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [February 5, 2022, 12:10pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/101 "2022-02-05T12:10:13Z")

</div>

As an update, TTFX for Blink.jl, Interact.jl, JSON.jl, CSV.jl and I would guess many other packages is largely dependent on the compilation time of Parsers.jl

[https://github.com/JuliaData/Parsers.jl/pull/108](https://github.com/JuliaData/Parsers.jl/pull/108)  
[https://github.com/JuliaIO/JSON.jl/pull/337](https://github.com/JuliaIO/JSON.jl/pull/337)

2200 dependencies means it’s probably responsible for a good fraction of the compilation time in the ecosytem.

My takeaway is not to fix your a packages precompile time, but the lowest level dependency that is causing the problem. ~~But it’s not always so easy to find where that is, because its buried so deep in the stack and usually doesn’t show up in the flame graph.~~ And to take the top of the `@snoopi_deep` graph more seriously! I could have found this a lot more quickly in hindsight.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [February 5, 2022, 9:30pm UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/102 "2022-02-05T21:30:22Z")

</div>

> [@Raf](#):
>
> But it’s not always so easy to find where that is, because its buried so deep in the stack and usually doesn’t show up in the flame graph.

Why would something not show up in the flame graph if it takes such a big chunk of the run time?

---

<div class="post-metadata">

**Author:** ![sdewaele](https://avatars.discourse-cdn.com/v4/letter/s/7ab992/32.png) [@sdewaele](https://discourse.julialang.org/u/sdewaele)\
**Post date:** [February 6, 2022, 5:00am UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/103 "2022-02-06T05:00:41Z")

</div>

While we should definitely continue to strive for lower TTFX, I think we should also clearly advertise the solution to use [PackageCompiler to create a sysimage](https://julialang.github.io/PackageCompiler.jl/stable/sysimages.html). It appears to me that this is a very practical and low effort method to eliminate TTFX. Of course creating a sysimage introduces a small inconvenience in the development workflow. However, I have found that in a typical project, the set of packages that I use stabilizes pretty early on, so there is no need to create a sysimage frequently. Also, importantly, it works perfectly for application deployment, where no new code is added.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [February 6, 2022, 7:59am UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/104 "2022-02-06T07:59:21Z")

</div>

Deeper functions dont show up because they’re taking compile time, not run time. So the flame graph will usually show functions further out that call the slow function, or nested functions that eventually call it.

Blink/Interact (partly via AssetRegustry) will show time in `JSON.Parser.parse`, and adding precompilation there does help some of the ttfx. But the real problem is in Parsers.jl, 5 method calls deeper. Its only used on one line to parse floats, and AssetRegistry doesn’t read or write floats. So its not exactly what you look for first, unless you know the packages already.

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [February 6, 2022, 8:15am UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/105 "2022-02-06T08:15:02Z")

</div>

PackageCompiler still takes some knowledge and the compile time problem is most important for the experience of newcomers.

If someone who uses R tries out Julia, they won’t be using package compiler straight away. So loading a tiny CSV and plotting a column from it will take half a minute for them, and Julia won’t feel at all “fast”.

And at the other end, as a developer, package compiler isn’t so useful.

---

<div class="post-metadata">

**Author:** ![sijo](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@sijo](https://discourse.julialang.org/u/sijo)\
**Post date:** [February 6, 2022, 9:01am UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/106 "2022-02-06T09:01:14Z")

</div>

I don’t think custom sysimages should be recommended for general use, before integration with Pkg is improved. See the “drawbacks” mentioned in the page you linked.

With a custom sysimage, as soon as you add a package to your project, you cannot trust Pkg (or the TOML files) anymore about which package versions you are using (you cannot even trust that the versions used are compatible).

It’s OK if you are disciplined and know what you are doing, but if people start using this without second thoughts… Imagine supposedly reproducible publications where it turns out the listed package versions are wrong because of this. This will hurt not only the researchers but also Julia’s reputation.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [February 6, 2022, 9:33am UTC](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949/107 "2022-02-06T09:33:43Z")

</div>

Ok then you mean the `@profile` flamegraph but not the `@snoopi_deep` flamegraph? I thought you were using the latter, it should show those methods I think

[Previous page](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949.md?page=4)

[Next page](https://discourse.julialang.org/t/taking-ttfx-seriously-can-we-make-common-packages-faster-to-load-and-use/74949.md?page=6)