# Compiling Julia using LTO+PGO
**URL:** https://discourse.julialang.org/t/compiling-julia-using-lto-pgo/39168
**Category:** Internals & Design
**Tags:** compilation
**Created:** [May 9, 2020, 9:10am UTC](https://discourse.julialang.org/t/compiling-julia-using-lto-pgo/39168 "2020-05-09T09:10:15Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@D4taV1s](https://discourse.julialang.org/u/D4taV1s)
#### Post date: [May 9, 2020, 9:10am UTC](https://discourse.julialang.org/t/compiling-julia-using-lto-pgo/39168/1 "2020-05-09T09:10:15Z")
to what extent would this affect performance, also for julia, what part outside of benchmark framework
isn’t measured e.g. not only throughput, latency(time to first paint, hurts), RAM/power usage, how close is this to being setup/built
-may more be of use to release builds
- [PGO](https://en.wikipedia.org/wiki/Profile-guided_optimization) requires a generate run, (how to improve),
- [LTO](https://www.llvm.org/docs/LinkTimeOptimization.html) has existed for years (less bugs,see gcc-6 [changelog](https://gcc.gnu.org/gcc-6/changes.html))
experiments (linux) much smaller binary sizes (dead code elimination, performance not much difference, can it be improved?)
- sys.so 186.6→137.5MB
- libjulia.so.1.4 32.3→5.6MB
but some not affected
- libLLVM-8jl.so 56.9→56.9MB
- libopenblas64\_.0.3.5.so 30.6→30.6MB
is there a better way than adding `-fprofile-generate` then `-fprofile-use` , `-O3 -march=native -flto` to (C,CXX,LD)FLAGS environment variables? + modern compiler (gcc8+),
Profiled (PGO) builds usually make use of a run that exercises the code for profiling (with representative coverage) (e.g. python compile has option)
non-expert, not much on @ certain programming (irrelevant?)
various links:
> **[Fedora 32 Aiming To Enable Link-Time Optimizations By Default For Packages](https://www.phoronix.com/news/Fedora-32-LTO-Packages)**
>
> In addition to finally enabling FSTRIM for flash-based storage devices, another arguably long overdue change slated for Fedora 32 to benefit performance is compiling packages by default with link-time optimizations (LTO) by the GCC compiler.
> **[Mesa Developers Discuss LTO'ing + PGO'ing Builds For Greater Performance](https://www.phoronix.com/news/Mesa-2020-PGO-LTO-Builds)**
>
> Making use of Link-Time Optimizations (LTO) and Profile Guided Optimizations (PGO) is currently being talked about by Mesa developers for their release builds in potentially squeezing out better performance.
> **[Even more fun with building and benchmarking Firefox with GCC and Clang](https://hubicka.blogspot.com/2018/12/even-more-fun-with-building-and.html)**
>
> Recent switch of official Firefox builds to Clang on all platforms has triggered some questions about the quality of code produced by GCC c...
some performance benchmarks:
> **[opensuse2018-e.pdf](https://www.ucw.cz/~hubicka/slides/opensuse2018-e.pdf)**
>
> 329.93 KB
why not/hinderances: (LTO seems to make debugging harder and PGO has to be compile twice (if profile genreation is unrepresentive there may be lack of improvement as result), thus ways to improve ease of use and only for release builds maybe esp for faster more emphasis on program that use up “CPU time”(effort)
but for people not building julia, a major point heard is the time-to-first-paint, would this have an impact, how much
notice LLVM parts don’t seem to be impacted, I don’t know enough about the build process to affect it (same size) is this of use ?.. [https://github.com/facebookincubator/BOLT/blob/master/docs/OptimizingClang.md](https://github.com/facebookincubator/BOLT/blob/master/docs/OptimizingClang.md)
---
### Author:  [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)
#### Post date: [November 14, 2021, 12:14pm UTC](https://discourse.julialang.org/t/compiling-julia-using-lto-pgo/39168/2 "2021-11-14T12:14:36Z")
I hope there will be an official “optimized” Julia binary for the **“x86\_64 feature levels”**
- `-O3 -march=x86-64-v3` +LTO +PGO +BOLT
- `-O3 -march=x86-64-v4` +LTO +PGO +BOLT
The x86\_64 feature levels [have been merged](https://www.phoronix.com/scan.php?page=news_item&px=LLVM-Clang-12-Microarch-Levels) to LLVM Clang 12
- **x86-64-v3:** (close to Haswell) AVX, AVX2, BMI1, BMI2, F16C, FMA, LZCNT, MOVBE, XSAVE
- **x86-64-v4:** AVX512F, AVX512BW, AVX512CD, AVX512DQ, AVX512VL
related julia issue:
- [https://github.com/JuliaLang/julia/issues/42073](https://github.com/JuliaLang/julia/issues/42073)
related news:
- [2021nov04: Facebook’s BOLT Nearing Mainline LLVM For Optimizing Binaries](https://www.phoronix.com/scan.php?page=news_item&px=BOLT-Nearing-LLVM-Monorepo)
Comment:
- The [“Pyston” (~ optimized Python)](https://www.pyston.org/) project started using BOLT optimization
- [https://github.com/pyston/pyston/pull/67](https://github.com/pyston/pyston/pull/67)
- “Tested that this is about 5.5% faster for kinto\_bench\_unopt”
- And _"all optimizations enabled (LTO+PGO) "_[with the default make](https://github.com/pyston/pyston#building)
---
### Author:  [@giordano](https://discourse.julialang.org/u/giordano)
#### Post date: [November 14, 2021, 2:34pm UTC](https://discourse.julialang.org/t/compiling-julia-using-lto-pgo/39168/3 "2021-11-14T14:34:19Z")
> [@ImreSamu](#):
>
> I hope there will be an official “optimized” Julia binary for the **“x86\_64 feature levels”**
Why do you think that’s important?
---
### Author:  [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)
#### Post date: [November 14, 2021, 4:11pm UTC](https://discourse.julialang.org/t/compiling-julia-using-lto-pgo/39168/4 "2021-11-14T16:11:16Z")
> [@giordano](#):
>
> Why do you think that’s important?
Benchmark would be better; so please correct me if I am wrong:
My reasons:
- **placebo & marketing**
- in the future the downloadable `-O3 -march=x86-64-v3(4) +LTO +PGO +BOLT` image will be a sign of dogfooding optimization.
- slimmer binary size expected ( vs current “X86: multi-microarchitecture system image” )
- ideal for the optimized Julia docker images
- now the X86 [“multi-microarchitecture system image”](https://github.com/JuliaLang/julia/blob/master/doc/src/devdocs/sysimg.md#specifying-multiple-system-image-targets) is
- `"generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)"`
- haswell=~x86-64-v3 OK ( but not visible for the first time users )
- And no optimized x86-64-v4
- “+LTO +PGO +BOLT” → faster compile time expected for the end users
- The “BOLT”-ed image is harder …
CON:
- more complexity
- need more build time. ( not a CI/CD friendly )
- need more test and developer resources
IMHO:
- it would be an interesting GSOC 2022(23) project for creating an **“+LTO +PGO +BOLT”** -ed Julia image. ( as an research project - helping energy-efficient green computing )
---
### Author:  [@stabbles](https://discourse.julialang.org/u/stabbles)
#### Post date: [November 24, 2021, 2:44pm UTC](https://discourse.julialang.org/t/compiling-julia-using-lto-pgo/39168/5 "2021-11-24T14:44:42Z")
The performance gain in LLVM is likely negligible. I tried compiling all julia’s dependencies with -march=zenvr2 using GCC 10 and then benchmarked precompile times of LLVM.jl a few times.
With generic binaries for Julia:
```julia
11.141739 seconds (1.94 M allocations: 133.629 MiB, 0.29% gc time, 6.16% compilation time)
11.106031 seconds (1.94 M allocations: 133.632 MiB, 0.19% gc time, 6.15% compilation time)
11.183070 seconds (1.94 M allocations: 133.614 MiB, 0.55% gc time, 5.84% compilation time)
11.084295 seconds (1.94 M allocations: 133.610 MiB, 0.55% gc time, 6.12% compilation time)
```
With -march=znver2:
```julia
10.917630 seconds (1.94 M allocations: 133.787 MiB, 0.30% gc time, 5.74% compilation time)
10.977101 seconds (1.94 M allocations: 133.803 MiB, 0.53% gc time, 5.79% compilation time)
11.000003 seconds (1.94 M allocations: 133.807 MiB, 0.38% gc time, 5.73% compilation time)
10.920701 seconds (1.94 M allocations: 133.804 MiB, 0.56% gc time, 6.12% compilation time)
```
If you want to try it yourself: [https://github.com/spack/spack/pull/27280#issue-1047361063](https://github.com/spack/spack/pull/27280#issue-1047361063).
```julia
taskset -c 0 ./spack/opt/spack/linux-sles15-zen2/gcc-10.3.0/julia-1.7.0-rc3-47wy4knrqrzqqga56jeau55epdl5mkvz/bin/julia -e 'using Pkg; @time Pkg.precompile()'
```
* * *
Edit: a slightly more interesting benchmark where some code is compiled and run. The following script:
```julia
using LoopVectorization
function f!(z, x, y)
@avx for i = eachindex(z)
z[i] = x[i] * y[i]
end
z
end
f!(rand(10), rand(10), rand(10))
```
with LoopVectorization 0.12.98 run as follows:
```julia
julia --project -e 'using Pkg; Pkg.instantiate(); @time include("script.jl")'
```
Generic binaries & sysimage:
```julia
13.497693 seconds (18.24 M allocations: 984.107 MiB, 2.38% gc time, 91.57% compilation time)
13.464525 seconds (18.24 M allocations: 984.137 MiB, 2.35% gc time, 91.43% compilation time)
13.513310 seconds (18.24 M allocations: 984.137 MiB, 2.58% gc time, 91.50% compilation time)
13.485646 seconds (18.24 M allocations: 984.135 MiB, 2.41% gc time, 91.38% compilation time)
```
-march=zenvr2:
```julia
12.997209 seconds (18.26 M allocations: 985.012 MiB, 2.45% gc time, 91.29% compilation time)
13.035375 seconds (18.26 M allocations: 985.014 MiB, 2.42% gc time, 91.15% compilation time)
13.014416 seconds (18.26 M allocations: 985.016 MiB, 2.44% gc time, 91.15% compilation time)
13.042701 seconds (18.26 M allocations: 985.014 MiB, 2.46% gc time, 91.10% compilation time)
```