# Does Debian's BenchmarkGames show representative performance?
**URL:** https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018
**Category:** Community
**Tags:** benchmark
**Created:** [April 25, 2022, 8:48am UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018 "2022-04-25T08:48:46Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@Volker\_Weissmann](https://discourse.julialang.org/u/Volker_Weissmann)
#### Post date: [April 26, 2022, 6:51pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/21 "2022-04-26T18:51:24Z")
The word “performance” is usually only used to describe speed of execution. Writeability is of course important, but no one uses the word “performance” for that.
---
### Author:  [@jar1](https://discourse.julialang.org/u/jar1)
#### Post date: [April 26, 2022, 6:53pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/22 "2022-04-26T18:53:33Z")
> [@Ahmed\_Salih](#):
>
> Do we happen to have any mesaures for performance instead in regards to writeability of code, number of lines etc.?
The “gz” numbers on the [benchmarks](https://benchmarksgame-team.pages.debian.net/benchmarksgame/performance/binarytrees-gz.html) reflect the source size, after a few simplifying transformations.
[source](https://twitter.com/erikengheim/status/1484594543211081728)
---
### Author:  [@igouy](https://discourse.julialang.org/u/igouy)
#### Post date: [April 26, 2022, 9:54pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/23 "2022-04-26T21:54:16Z")
> [@jar1](#):
>
> Setting up benchmarking infrastructure is a significant job…
That seems to have been [done already](https://github.com/JuliaCI/BenchmarkTools.jl).
Seems to me that the Julia community have every opportunity to make and share whatever detailed comparisons might interest them.
---
### Author:  [@acxz](https://discourse.julialang.org/u/acxz)
#### Post date: [April 27, 2022, 1:05am UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/24 "2022-04-27T01:05:20Z")
In terms of bench-marking infrastructure across languages see [JuliaLang/Microbenchmarks](https://github.com/JuliaLang/Microbenchmarks).
---
### Author:  [@Elrod](https://discourse.julialang.org/u/Elrod)
#### Post date: [April 27, 2022, 1:23am UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/25 "2022-04-27T01:23:19Z")
> [@ChrisRackauckas](#):
>
> The one major issue with the benchmark games is its inconsistent with its rules around compilation. For C, Fortran, etc. it doesn’t measure the compilation time. With Julia it does. These codes are simple enough that more are even compatible with StaticCompiler.jl, and all are compatible with PackageCompiler.jl, so they are able to build a .so binaries to be ran just like C or Fortran, but for some reason that’s forbidden for just Julia?
There are differences in how the languages are used, so if you want to reflect typical use cases, most consumers of C/C++/Fortran/Rust code don’t sit through compilation.
But developers do.
One could also argue that Julia should be allowed to use `-march=native` (the default) because this is how it’s generally used, but that the other languages should not be, as they’re generally compiled with generic targets and distributed.
Of course, most code does a lousy job taking advantage of the hardware so that `-march=native` rarely makes much of a difference, and then the code that is optimized (like BLAS) does ship multiple versions…
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [April 27, 2022, 2:59am UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/26 "2022-04-27T02:59:36Z")
> [@Elrod](#):
>
> There are differences in how the languages are used, so if you want to reflect typical use cases, most consumers of C/C++/Fortran/Rust code don’t sit through compilation.
> But developers do.
And we do match that. The JIT compilation is seen in developer space but in deployments we do system image construction and such before putting Julia into the cloud, airplanes, and other real hardware. What seems to be the more interesting benchmark these days isn’t the interactive development but the speed of the compiled deployments and how much compilation can be removed from that.
---
### Author:  [@AMJ](https://discourse.julialang.org/u/AMJ)
#### Post date: [April 27, 2022, 6:41am UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/27 "2022-04-27T06:41:46Z")
Julia in airplanes? My impression was that we could not do hard real time with Julia, or is it for something not latency insensitive?
---
### Author:  [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)
#### Post date: [April 27, 2022, 7:11am UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/28 "2022-04-27T07:11:24Z")
[ ](https://www.youtube.com/watch?v=19zm1Fn0S9M)
---
### Author:  [@jar1](https://discourse.julialang.org/u/jar1)
#### Post date: [April 27, 2022, 8:25am UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/29 "2022-04-27T08:25:09Z")
That’s using Julia as a specification format and simulation tool, not to actually run on aircraft.
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [April 27, 2022, 12:20pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/30 "2022-04-27T12:20:16Z")
> [@AMJ](#):
>
> Julia in airplanes? My impression was that we could not do hard real time with Julia, or is it for something not latency insensitive?
I can’t give more details of this specific one. But no, you can just ship binaries without compilation latency. This is Julia of 2022, we’re doing this in many applications with PackageCompiler and StaticCompiler. Using the JIT interface is really just a developer interface and generating compiled and/or static code is something we (Julia Computing) are doing daily.
@carstenbauer that’s not related.
---
### Author:  [@jbu](https://discourse.julialang.org/u/jbu)
#### Post date: [April 27, 2022, 12:50pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/31 "2022-04-27T12:50:24Z")
> [@ChrisRackauckas](#):
>
> Using the JIT interface is really just a developer interface
Huh, this is news to me. As a fairly novice Julia programmer, I thought the JIT interface was the primary way to program in Julia.
I’ve been aware of StaticCompiler.jl and PackageCompiler.jl for some time, but neither seemed to be production-ready yet.
Are there resources for how to incorporate a binary-based/static code workflow into Julia for, e.g., reducing TTFX?
---
### Author:  [@lmiq](https://discourse.julialang.org/u/lmiq)
#### Post date: [April 27, 2022, 1:17pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/32 "2022-04-27T13:17:45Z")
If TTFX is very inconvenient to you, while developing, you can try it:
> [@A Julia DataAnalysis Sysimage from PackageCompiler It's so easy you should do it too!](https://discourse.julialang.org/t/a-julia-dataanalysis-sysimage-from-packagecompiler-its-so-easy-you-should-do-it-too/68127):
>
> If you want to use Julia for the kind of thing where you might fire up R and read a couple CSV files into some DataFrames, maybe grab some data from a SQLite file, manipulate the data a little, make a few plots, and be done… Then it’d be nice to have a quick-load sysimage so you don’t have too much “time to first plot”. It turns out that PackageCompiler has gotten to the point where this is fairly trivial. Here are two scripts I’m using to build my own sysimage that can do all these things, incl…
But, IMHO, it is more convenient to deal with that using a Revise-based worflow. I would not recommend start building system images to everyone.
At the end you could deliver a binary for _users_ of your package, without the TTFX at all (and that could be used for benchmarking in some contexts). But I personally think that is a necessity that first needs to be demonstrated in the specific domain/application in question.
Concerning the original question of the post, my opinion is that “performance” and " development responsiveness" are two different things. Of course there is a TTFX in Julia that you may or may not include in the benchmark. Yet, I would not call that a measure of “performance”, at least in my field, things that take a few seconds to run are not important, and we care about performance for things that take hours and days to run, for which TTFX and compilation are irrelevant.
---
### Author:  [@jbu](https://discourse.julialang.org/u/jbu)
#### Post date: [April 27, 2022, 1:46pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/33 "2022-04-27T13:46:59Z")
Thanks for that great resource! I’ll give PackageCompiler another shot.
> [@lmiq](#):
>
> But, IMHO, it is more convenient to deal with that using a Revise-based worflow.
Oh yes, Revise is awesome 😀 I use it on a daily basis, and it’s in my `startup.jl`. But when developing a package requiring frequent reboots of the REPL (e.g. changing structs, or clearing obsolete function method definitions), you still have to deal with large delays. For my workflow this can be on the order of 30 seconds to ~2+ minutes for each reboot (with Julia 1.7.2).
> [@lmiq](#):
>
> At the end you could deliver a binary for _users_ of your package, without the TTFX at all (and that could be used for benchmarking in some contexts). But I personally think that is a necessity that first needs to be demonstrated in the specific domain/application in question.
Binaries are the default for robotics, autonomous systems, and embedded systems. That’s why I’m interested in any developments on static compilation with Julia. There’s a few pain points remaining, but I’m optimistic that options for Julia binaries will keep improving 🙂.
> [@lmiq](#):
>
> Of course there is a TTFX in Julia that you may or may not include in the benchmark. Yet, I would not call that a measure of “performance”, at least in my field, things that take a few seconds to run are not important, and we care about performance for things that take hours and days to run, for which TTFX and compilation are irrelevant.
Completely agree that compilation time doesn’t matter for benchmarking final code. As an aside though, I think responsiveness and TTFX are key aspects of _development speed_ which is also an important aspect to consider. This has been discussed a lot though, so I won’t go down that rabbit hole here.
---
### Author:  [@AMJ](https://discourse.julialang.org/u/AMJ)
#### Post date: [April 27, 2022, 7:31pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/34 "2022-04-27T19:31:38Z")
Great. I meant garbage collection related latencies and/or stuff like grantees for certain amount of executions under a certain time. My last thinking was you should be able to do soft real time as long as you don’t allocate, how true/hard is this as of Julia 2022?
---
### Author:  [@suavesito](https://discourse.julialang.org/u/suavesito)
#### Post date: [April 27, 2022, 7:41pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/35 "2022-04-27T19:41:56Z")
Surely you can do hard realtime even allocating _if_ you use [StaticTools.jl](https://github.com/brenhinkeller/StaticTools.jl) and [StaticCompiler.jl](https://github.com/tshort/StaticCompiler.jl).
---
### Author:  [@AMJ](https://discourse.julialang.org/u/AMJ)
#### Post date: [April 27, 2022, 7:59pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/36 "2022-04-27T19:59:52Z")
Ok, pardon me if my question seems a little ignorant since Im not knowledgeable enough in this area, but assuming StaticCompiler/StaticTools/Jet… gets stable enough then what advantage do languages like C++/Rust hold over Julia other than making programmers life harder?
---
### Author:  [@suavesito](https://discourse.julialang.org/u/suavesito)
#### Post date: [April 27, 2022, 8:09pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/37 "2022-04-27T20:09:38Z")
I would argue (minor caveats) that StatiCompiler + StaticTools are “stable enough” for some things in production, see [this thread](https://discourse.julialang.org/t/successful-static-compilation-of-julia-code-for-use-in-production/79318). They will get even better over time, but they’re usable right now.
And answering the question, Rust/C++ and other languages have other sets of features _and_ target objectives, like interfaces for Rust and C++, memory safety for Rust, and the case that both languages are more guided to systems programming. Those features _do_ make them more useful for some task, so Julia will not replace them on what they do well. 🙂
---
### Author:  [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)
#### Post date: [April 27, 2022, 8:32pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/38 "2022-04-27T20:32:54Z")
> [@jbu](#):
>
> I’ve been aware of StaticCompiler.jl and PackageCompiler.jl for some time, but neither seemed to be production-ready yet.
> [@AMJ](#):
>
> Great. I meant garbage collection related latencies and/or stuff like grantees for certain amount of executions under a certain time. My last thinking was you should be able to do soft real time as long as you don’t allocate, how true/hard is this as of Julia 2022?
They aren’t perfect, but Julia computing is already building binaries are parts of products. For example, the FMU Accelerator.
[ ](https://www.youtube.com/watch?v=ws31EAHwpmk)
And that’s just the most public thing we have on this. We ship all things on the cloud as system images as well, and then many of the cases where people take the simulations and bring them to actual devices uses PackageCompiler/StaticCompiler (again, cannot share too many details on this part, but some may share at JuliaCon!)
---
### Author:  [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [April 27, 2022, 8:46pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/39 "2022-04-27T20:46:36Z")
> [@jbu](#):
>
> I’ve been aware of StaticCompiler.jl and PackageCompiler.jl for some time, but neither seemed to be production-ready yet.
I believe PackageCompiler.jl is production-ready, I believe (1.0 came out in Feb 2020 and 2.0 in Nov 2021, now at 2.0.5), for applications, at least used in prod (likely by e.g. for PumasAI company, I guessed what Chris is referring to, but see he mentioned for other stuff).
I do not know of any official statements for any [Julia] package. I suppose it may be inferred when 1.0 version is released, but a lot of code has been used in production before that, including Julia; and e.g. JuMP.jl at 10 years old only recently hit 1.0, was state-of-the-art (and I suppose used) years ago.
I understand you need to be careful when building a sysimage, know the limitations (package manager will not update the packages in it); also in general, especially with experimental software, which I do not know it to be.
StaticCompiler.jl is pre-1.0, and I see “experimental”, but neither has stopped [all] people from using such code in production (e.g. Julia itself pre-1.0, or its Threads API marked “experimental”, back in 0.5 until 1.5 in 2020).
Despite all that, I see (and also Chris mentioning it used in production):
> [@Successful Static Compilation of Julia Code for use in Production](https://discourse.julialang.org/t/successful-static-compilation-of-julia-code-for-use-in-production/79318):
>
> The [StaticCompiler](https://github.com/tshort/StaticCompiler.jl) package was recently registered. This post records a successful experiment to statically compile a piece of Julia code into a small .so library on Linux, which is then loaded from Python and used in training of a deep learning model. TLDR Static compilation to a stand-alone library does work on Linux but has rather significant restrictions on the functionality available. It is mostly useful for core computational routines. Roughly speaking, if it could be implemented in plai…
Anyone can make a code-generator in Julia, and that’s also one way, I noticed for (this) restrictive (seemingly?) scenario, possible to compile Julia to C, MATLAB and more:
>
---
### Author:  [@jbu](https://discourse.julialang.org/u/jbu)
#### Post date: [April 28, 2022, 1:49pm UTC](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018/40 "2022-04-28T13:49:33Z")
> [@ChrisRackauckas](#):
>
> They aren’t perfect, but Julia computing is already building binaries are parts of products. For example, the FMU Accelerator…And that’s just the most public thing we have on this. We ship all things on the cloud as system images as well, and then many of the cases where people take the simulations and bring them to actual devices uses PackageCompiler/StaticCompiler
> [@Palli](#):
>
> I believe PackageCompiler.jl is production-ready, I believe (1.0 came out in Feb 2020 and 2.0 in Nov 2021, now at 2.0.5), for applications, at least used in prod
That’s fantastic that Package/StaticCompiler are getting that level of usage! I’m excited to see what future developments are in store for those packages.
[Previous page](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018.md?page=1)
[Next page](https://discourse.julialang.org/t/does-debians-benchmarkgames-show-representative-performance/80018.md?page=3)