# Permanent recompilation of libraries and performance issues
**URL:** https://discourse.julialang.org/t/permanent-recompilation-of-libraries-and-performance-issues/90815
**Category:** General Usage
**Created:** [November 25, 2022, 3:56pm UTC](https://discourse.julialang.org/t/permanent-recompilation-of-libraries-and-performance-issues/90815 "2022-11-25T15:56:58Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Michael\_Hecht](https://discourse.julialang.org/u/Michael_Hecht)
#### Post date: [November 25, 2022, 3:56pm UTC](https://discourse.julialang.org/t/permanent-recompilation-of-libraries-and-performance-issues/90815/1 "2022-11-25T15:56:58Z")
I have a little test program that uses some libraries
```julia
println("load libs ...")
t0 = time()
@time using JuMP
@time using Ipopt
println("run prog ...")
m = Model(Ipopt.Optimizer)
println("set variables ...")
@variable(m,x[1:2])
@NLobjective(m,Min,(x[1]-3)^3+(x[2]-4)^2+x[1]*x[2])
@NLconstraint(m,(x[1]-1)^2+(x[2]+1)^3+exp(-x[1])<=1)
println(m)
println("optimize ...")
@time JuMP.optimize!(m)
println("** Optimal objective function value = ",JuMP.objective_value(m))
println("** Optimal solution = ",JuMP.value.(x))
println()
println("prog took ",round(time()-t0,digits=2)," sec ...")
```
On every start of the program (I did not change anything and I did not touch Pkg) it takes several seconds for recompilation:
```julia
load libs ...
5.880101 seconds (8.24 M allocations: 636.694 MiB, 6.29% gc time, 8.85% compilation time: 97% of which was recompilation)
1.396453 seconds (1.36 M allocations: 72.658 MiB, 10.58% gc time, 91.54% compilation time: 100% of which was recompilation)
```
This cannot be a correct behavior, is it?
The program is overall very slow.
```julia
prog took 23.39 sec ...
```
I wonder if there is something wrong with my Julia installation (Windows 11, fast and modern computer, SSD, CPU far below 100%).
---
### Author:  [@j-fu](https://discourse.julialang.org/u/j-fu)
#### Post date: [November 25, 2022, 4:18pm UTC](https://discourse.julialang.org/t/permanent-recompilation-of-libraries-and-performance-issues/90815/2 "2022-11-25T16:18:08Z")
Essentially this is to be expected, as Julia caches an intermediate representation over different sesssions, and by its dynamic nature it needs to be aware that methods may be called with different types and thus compiled for specialization. There is serious work underway in Julia proper and in many packages to improve the situation, in particular by caching machine code. You can find more (and more precise) information in [this Juliacon talk](https://live.juliacon.org/talk/DUQQLN). It is expected to see the results in one of the next releases.
That said, there are some general workflow tipps which can drastically improve your turnaround time just now. I did a writeup on this [here](https://j-fu.github.io/marginalia/julia/basic-workflow/).
---
### Author:  [@Michael\_Hecht](https://discourse.julialang.org/u/Michael_Hecht)
#### Post date: [November 25, 2022, 4:48pm UTC](https://discourse.julialang.org/t/permanent-recompilation-of-libraries-and-performance-issues/90815/3 "2022-11-25T16:48:10Z")
Can you test the program? What time takes it on your computer? If I remove the about 7 seconds of library loading, it is finally very slow without using much CPU time.
EDIT:
Checked the hints on your link. Loading the script in the REPL and than calling main leads to execution time of 0.09 sec. So the question would be why Julia isn’t able to remember the compiled version (loading libraries cannot be cached?!) via a cache.
Working only from REPL seems to me quite inconvenient if you are used to start a program from shell …
---
### Author:  [@j-fu](https://discourse.julialang.org/u/j-fu)
#### Post date: [November 25, 2022, 5:32pm UTC](https://discourse.julialang.org/t/permanent-recompilation-of-libraries-and-performance-issues/90815/4 "2022-11-25T17:32:29Z")
I came from C++/python and had to take this hurdle as well… How much starting from shell is important for you is of course a matter of personal preferences, for me the advantages outweigh this problem.
With your posted code I get
```julia
prog took 12.43 sec
```
Wrapping into a module and a function gives on an Intel(R) Core™ i7-9850H CPU @ 2.60GHz
```julia
prog took 7.09 sec
```
for the first run and
```julia
prog took 0.02 sec
```
for the second. So this is consistent to your observations.
To get an impression on the ongoing work on saving compiled machine code, besides the video you can investigate
>
>
> \# Overview
>
> This aims to establish the foundation needed to save native code i…n "package images." From a technical standpoint, all it does is replace our current mix of serializers (\`dump.c\` for \`\*.ji\` files, \`staticdata.c\` for system images) with a unified serializer based on \`staticdata.c\`. Necessary functionality like uniquing types and MethodInstances, supporting external CodeInstances and new method roots, and load-time invalidations are now supported by \`staticdata.c\`. A key feature is the ability to link externally: the serialization format defines a tag for an external object, which is linked after loading by pointer relocation.
>
> The core system image and all "package images" are stored as contiguous blobs in memory, each identified by a pair of \`\*begin, \*end\` pointers stored in \`jl\_linkage\_blobs\`. Each of these images is identified by the module \`build\_id\`, and this value is used in encoding external references. For individual objects, "ownership" is decided by pointer address, i.e., which pointer pair encloses the given object. Some objects, like \*external\* MethodInstances (new specializations of callables owned by other packages), are deliberated created with pointer addresses that fall out of these ranges to ensure that they go through the uniquing pipeline.
>
> Compared to the original version of this PR, we've temporarily stripped all the work devoted to native code support. That will be restored in a future PR expected to land shortly after this merges. The goal here is to transition to one-serializer-to-rule-them-all without breaking Julia. It completes the goal of the original PR and items 2 & 4 listed in the "Future work (TODOs)". Item 3 is already merged to master, so overall it has become far more ambitious than its original scope despite having taken a step backwards with respect to native code.
>
> This has become a 3-way collaboration (@vchuravy, @vtjnash, and @timholy), with the recent participation of @vtjnash having added enormously to our progress.
>
> For reference, the original version of this post is included below, but note that several points no longer apply.
>
> \-------------------
>
> \# Overview (original)
>
> This pull request is the first fruit of a tight collaboration between @vchuravy and @timholy. Our hope is that this is the inaugural PR in a series whose ultimate goal is to allow packages to save & reuse their precompiled native code. A second outcome might be to enable (or contribute to enabling) StaticCompiler.jl to be implemented with few external dependencies.
>
> The journey is long, and this first pull request is intended to have no user-visible consequences (neither good nor bad). But we believe it establishes many of the necessary fundamentals.
>
> \# Background
>
> Caching code requires \[serialization\](https://en.wikipedia.org/wiki/Serialization) and deserialization. Julia has several code (de)serializers, but here our focus is on two, the one in \`dump.c\` and the one in \`staticdata.c\`. \`dump.c\` writes the \`.ji\` files that we currently use for packages and an intermediate stage of building Julia, whereas \`staticdata.c\` creates the object files (\`.so\` on Linux) that serve as Julia's system image. \`dump.c\` can (now, after #43990) save almost all supportable objects \*except\* native code. Conversely, \`staticdata.c\` can save native code and has a more streamlined design, but currently is only useful for writing monolithic system images. Part of the ultimate goal of the series of PRs is to blend the best of both (de)serializers together.
>
> In addition, substantial changes will be required in Julia's codegen/LLVM infrastructure. The core issue is that caching native code across multiple files is a lot like building a C application from a bunch of separate \`.o\` files: you need a linking step to get them to work together. Currently, Julia has no real mechanisms to perform this linking. Interaction among packages has to persist after the LLVM modules used to assemble them have been discarded.
>
> \# Details
>
> The only way to engage the functionality here is to launch Julia with command-line arguments
>
> \`\`\`
> \--output-o $output --output-incremental=yes
> \`\`\`
>
> which is not a supported combination on \`master\`. Hence this will \*not\* be used in package precompilation until we switch \`base/loading.jl\` to issue this combination of command-line arguments. Consequently, we can develop the required infrastructure, get all the pieces working, and then turn it on for regular usage.
>
> This first PR aims to allow system-image-like blobs ("package images"?) to encode \*external references\* and perform most of the necessary linking to connect them. It consists of a new "tagged" serialization enum, \`ExternalLinkage\`, used to encode these links. It also includes much of the LLVM functionality needed for successful linkage, with one major exception described below.
>
> Fundamentally, cross-references among Julia internal structures are made by pointers. External linkage is therefore achieved by pointer relocation. During serialization, package contents are copied into a single contiguous blob of memory. To make pointers relocatable, an external reference is decomposed into two pieces:
>
> \- to encode "which blob are we linking against?", we use the \`build\_id\` of the toplevel module in the precompilation \`worklist\`
> \- within a blob, identity is determined by the offset from the blob's base pointer.
>
> An introduction to details of the (de)serialization mechanisms used in \`staticdata.c\` can be found in the extensive comment at the top of the file.
>
> \## What this does
>
> This successfully serializes and deserializes external links, and implements much of the functionality needed for "partial" LLVM modules. In particular, we support:
>
> \- saving lowered, type-inferred, and native code for methods defined in the package
> \- accessing global variables from the same package, even from compiled code
> \- calling compiled functions in other package images (partial, see below)
> \- accessing global variables defined in other package images, even from compiled code
>
> It also introduces a "stub" implementation of new standard library, \`LLD\_jll\`, used for performing some of the linkage.
>
> \## What this doesn't do
>
> Currently, the decision about whether a reference is internal vs external is deliberately over-simplified, and arrives at the wrong answer in important cases, such as when PkgB triggers novel specialization of a method in PkgA. Because of some challenges involving exported names and/or the need for a \[trampoline\](https://en.wikipedia.org/wiki/Trampoline\_%28computing%29), it also duplicates native code in the \`.text\` section of the object file (written by LLVM) rather than linking to a unique implementation.
>
> \## Future work (TODOs)
>
> We expect this PR to be followed by at least four more PRs:
>
> 1. one that implements de-duplication of the native code (likely via implementation of a trampoline)
> 2. one that expands/migrates functionality from \`dump.c\` to \`staticdata.c\` (adding methods to external functions, compiling novel specializations of external methods, uniquing compilations of the same \`MethodInstance\` by multiple downstream packages, managing backedges and invalidation, etc.)
> 3. one that makes \`LLD\_jll\` a "real" standard library
> 4. one that makes this the default (or only) mechanism for precompiling packages
>
> We welcome participation by others in these future developments.
>
> \## Future prospects
>
> If all this works as well as we hope, we expect to see dramatic decreases in latency for precompiled workloads. Indeed, in favorable cases with little or no invalidation, compilation time may be nearly eliminated.
>
> In such cases, the majority of Julia's remaining latency problem will be due to package loading. We do not expect this sequence of PRs to make load times worse, but despite efficiencies in the \`staticdata.c\` representation we also don't expect there to be much improvement: raw deserialization is likely to become faster, but on current master it is already dominated by the cost of method insertion and invalidation, and that won't change in this sequence of PRs. In the future, there are a number of possible ways to improve load times (perhaps dramatically), but we plan to get this whole series merged first before even beginning to contemplate tackling load times.
>
> \## Ideal schedule
>
> We're well aware that important work remains to finalize Julia 1.8, and that work should take priority. However, once that ramps down it would be great to get this reviewed and merged fairly early in the 1.9 cycle. There's a long ways yet to go, and we'll need time if we are to get the entire sequence merged for 1.9.
---
### Author:  [@Michael\_Hecht](https://discourse.julialang.org/u/Michael_Hecht)
#### Post date: [November 26, 2022, 6:25am UTC](https://discourse.julialang.org/t/permanent-recompilation-of-libraries-and-performance-issues/90815/5 "2022-11-26T06:25:32Z")
Did you run it from REPL or from shell?
Putting the Code in a function also results in 12 sec for my computer (running from shell, 7 sec for loading libraries).
By the way: I’m coming from Python so I’m used to run always from shell without delay.
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [November 26, 2022, 7:37am UTC](https://discourse.julialang.org/t/permanent-recompilation-of-libraries-and-performance-issues/90815/6 "2022-11-26T07:37:33Z")
What you are running into is known as “TTFX”, “Time To First X” problem. There are loads of discussions on this on here and across the internet, but the short answer is that if you’re trying to use Julia as a scripting language for small programs you are going to be disappointed at this point, as startup and compilation time will entirely dominate your experience.
For that reason, (probably) 99% of Julians have a workflow which relies on a long running REPL session which means TTFX cost is only incurred once for each type-specific method call. This doesn’t necessarily mean you are actually interacting with the REPL, as lots of front ends are available (VSCode, Jupyter, Pluto notebooks, plug ins for other editors).
If you absolutely have to repeatedly call Julia from the shell, there’s
> **[GitHub - dmolina/DaemonMode.jl: Client-Daemon workflow to run faster scripts...](https://github.com/dmolina/DaemonMode.jl)**
>
> Client-Daemon workflow to run faster scripts in Julia - GitHub - dmolina/DaemonMode.jl: Client-Daemon workflow to run faster scripts in Julia
Targeted at this use case (I have never used it so am not sure it fits the bill for what you’re doing).
If you want to use Julia to build an executable that can be called externally there is also tooling for that in PackageCompiler (full language support, large binaries) and StaticCompiler (experimental, support for a very restricted subset of the language at this point, small binaries).
This is the situation as it stands, but as mentioned above there is a lot of work in this area at the minute so expect improvements in either Julia 1.9 or 1.10 (or both!)
---
### Author:  [@Michael\_Hecht](https://discourse.julialang.org/u/Michael_Hecht)
#### Post date: [November 26, 2022, 9:06am UTC](https://discourse.julialang.org/t/permanent-recompilation-of-libraries-and-performance-issues/90815/7 "2022-11-26T09:06:43Z")
Thank you all for the information. I really like Julia an will try to work around such inconveniences.
EDIT:
I tried DaemonMode.jl and this is really fast. I think I go with this until Julia is faster at this point.