# Julia position in the Debian Benchmark Game can be improved, and categorization of some Julia there is unfair
**URL:** https://discourse.julialang.org/t/julia-position-in-the-debian-benchmark-game-can-be-improved-and-categorization-of-some-julia-there-is-unfair/122280
**Category:** Performance
**Created:** [November 5, 2024, 11:22am UTC](https://discourse.julialang.org/t/julia-position-in-the-debian-benchmark-game-can-be-improved-and-categorization-of-some-julia-there-is-unfair/122280 "2024-11-05T11:22:05Z")
**Posts on this page:** 10
**Page:** 2
### Author:  [@non-Jedi](https://discourse.julialang.org/u/non-Jedi)
#### Post date: [November 7, 2024, 6:27pm UTC](https://discourse.julialang.org/t/julia-position-in-the-debian-benchmark-game-can-be-improved-and-categorization-of-some-julia-there-is-unfair/122280/21 "2024-11-07T18:27:59Z")
> [@jakobnissen](#):
>
> This specific rule is selectively applied.
I don’t think this is fair. I don’t necessarily agree that the rules Isaac (@igouy in this thread) has set out for the benchmarks game are the most interesting set of rules and [have argued with him a little bit on this exact issue](https://salsa.debian.org/benchmarksgame-team/benchmarksgame/-/issues/167), but he’s always been consistent in how he applies the rules. Just above the bit you quote, the [binary trees directions](https://benchmarksgame-team.pages.debian.net/benchmarksgame/description/binarytrees.html) say:
> When possible, use default GC; otherwise use per node allocation or use a library memory pool.
He’s clearly drawing a line between “library” memory pools and “custom” (inline) memory pools. Languages without a default GC are explicitly allowed to use the former but not the latter while languages with default GC aren’t allowed “library” nor “custom” memory pools.
> [@sdanisch](#):
>
> I’ve invested quite some time years ago to match C performance, which wasn’t that hard by just translating the C versions, but those versions never made it into the game for different reasons, which all felt unfair, considering they where 1:1 translations of the C versions that take first spot, seemingly without applying the same penalties.
Not sure if there was a different effort I wasn’t aware of, but I don’t think this is generally true of the ones in the [BenchmarksGame.jl](https://github.com/JuliaPerf/BenchmarksGame.jl) repo. As far as I know, all the fastest Julia implementations in the benchmarks game are now as fast or faster than those in that repo with two exceptions:
1. Binary trees, which implemented a custom memory pool rather than using default GC. The reason this isn’t allowed and whether it should be allowed has been discussed several times both on this forum and in the benchmarks game issue tracker.
2. Mandelbrot, which used 32-bit floats while all the other implementations were using 64-bit floats and [so couldn’t be accepted since it gives wrong results](https://salsa.debian.org/benchmarksgame-team/benchmarksgame/-/issues/135).
> [@Benny](#):
>
> I suppose no competitor knew their way around [SIMD.jl](https://juliahub.com/ui/Packages/General/SIMD).
Unfortunately at the time I was contributing to the benchmarks game, the extra load time from using an external package had a huge effect on total runtime since the game explicitly includes JIT compilation, so it was never worthwhile . This might no longer be the case with recent Julia versions.
> [@Mason](#):
>
> By the way, incidentally related to this, I recently played around with making some of these benchmarks run with julia’s new AOT compiler mode, and the results are quite promising: [GitHub - MasonProtter/juliac-bench](https://github.com/MasonProtter/juliac-bench)
👀!
---
### Author:  [@nsajko](https://discourse.julialang.org/u/nsajko)
#### Post date: [November 7, 2024, 7:32pm UTC](https://discourse.julialang.org/t/julia-position-in-the-debian-benchmark-game-can-be-improved-and-categorization-of-some-julia-there-is-unfair/122280/22 "2024-11-07T19:32:15Z")
> [@non-Jedi](#):
>
> while languages without default GC aren’t allowed “library” nor “custom” memory pools.
I think you meant “with” here, not “without”.
In any case, the point stands that some languages, those with a default GC, are handed a different problem to solve than other languages, those without a default GC.
---
### Author:  [@non-Jedi](https://discourse.julialang.org/u/non-Jedi)
#### Post date: [November 7, 2024, 7:50pm UTC](https://discourse.julialang.org/t/julia-position-in-the-debian-benchmark-game-can-be-improved-and-categorization-of-some-julia-there-is-unfair/122280/23 "2024-11-07T19:50:49Z")
Oops. Thanks. I’ll edit so later readers don’t get confused.
---
### Author:  [@sdanisch](https://discourse.julialang.org/u/sdanisch)
#### Post date: [November 7, 2024, 11:12pm UTC](https://discourse.julialang.org/t/julia-position-in-the-debian-benchmark-game-can-be-improved-and-categorization-of-some-julia-there-is-unfair/122280/24 "2024-11-07T23:12:50Z")
To add to that, I think it’s pretty unique, that Julia can so easily avoid the GC and make a memory pool in so few lines.
I have no idea what goal could be reached by putting a penalty on that - especially since there are already tons of different versions for the same language and problems, with threads, with simd, without etc… why not permit a version with gc and one without?
Anyways, don’t get me wrong, the rules are clear and not unfairly applied, but I do wonder why the Julia community should put any time in such a comparison, where there’s no chance to break even with the fast languages whatsoever, even though Julia can mostly reach the same performance 🤷
Juliac will be interesting to at least get rid of the compilation penalty, but then it still stands that Julia isn’t allowed to do the same optimisations as other languages.
---
### Author:  [@Benny](https://discourse.julialang.org/u/Benny)
#### Post date: [November 8, 2024, 1:23am UTC](https://discourse.julialang.org/t/julia-position-in-the-debian-benchmark-game-can-be-improved-and-categorization-of-some-julia-there-is-unfair/122280/25 "2024-11-08T01:23:21Z")
> [@sdanisch](#):
>
> I think it’s pretty unique, that Julia can so easily avoid the GC and make a memory pool in so few lines.
For that specific benchmark, I think most languages can make an adequate memory pool within a reasonably sized file. I can only hazard a guess that the intent of differentiating “library” vs “custom” allocation strategies is that libraries are available to anyone and designed to handle more problems than one benchmark.
> [@sdanisch](#):
>
> why the Julia community should put any time in such a comparison, where there’s no chance to break even with the fast languages whatsoever
Could at least compete with the other GC languages. I agree on separate benchmarks, but it’s not too difficult a caveat to filter out the non-GC-so-import-whatever languages. Earlier a Haskell entry was mentioned as using GHC.Compact to evade its GC, it’s interesting to note that it’s not much of an improvement of the next lower Haskell entry that is documented to precede that optimization. So after wading through the caveats of a GC benchmark, Julia has room to improve.
---
### Author:  [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [November 14, 2024, 4:29pm UTC](https://discourse.julialang.org/t/julia-position-in-the-debian-benchmark-game-can-be-improved-and-categorization-of-some-julia-there-is-unfair/122280/26 "2024-11-14T16:29:40Z")
[https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/binarytrees-java-7.html](https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/binarytrees-java-7.html)
Why is Java here 2.7x faster than Julia? It doesn’t seems cheating, i.e. not using memory pools (is using a thread pool that is ok; Julia also uses threads). All faster programs, i.e. the non-GC languages, use memory pools.
(Lisp and) OCaml aren’t far behind, and it has a comment:
> (\* Gc.set { (Gc.get()) with Gc.minor\_heap\_size = 1024 \* 1024; max\_overhead = -1; }; \*)
It’s actually a hint to the GC, and maybe Java is similarly good without needing a hint. Can Julia match this (i.e. without a memory pool)? Feel free to also implement with a pool.
~~I’m unsure, does~~ Java (and seemingly C#) allow heap compaction (I’m unsure if this is still used in practice, only in old GC implementations?)? Does it make it faster? I can’t actually see how it can be faster, if it moves memory around (that’s an overhead), which is not possible in Julia (since it must be able to call C?), but why was that ever done historically at least? If not for performance reasons?
[https://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html](https://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html)
> Step 2a: Deletion with Compacting
> To further improve performance, in addition to deleting unreferenced objects, you can also compact the remaining referenced objects. By moving referenced object together, this makes new memory allocation much easier and faster.
I’m unclear on how this can work at all without changing all pointers to the still-used regions, and indirection doesn’t seem like would help for performance. Pointers aren’t used much in Julia, but you can get pointers to heap objects, e.g. for calling C, so it rules this out. Is there a way to support it when you know pointers not needed?
All the faster GC-langugages there seem to have or support compacting GC (Java is using its default GC, one of many, haven’t confirmed it’s compacting).
Haskell is even faster, 3.4x Julia’s time, but it opts into some non-GC tracked memory, unsure if it’s compacting, seems more like a memory pool.
[https://www.reddit.com/r/haskell/comments/fxr4oj/ghccompact\_and\_the\_new\_gc/](https://www.reddit.com/r/haskell/comments/fxr4oj/ghccompact_and_the_new_gc/)
We’re 47% slower than even Erlang, and it compacts (unless no longer using such GC).
Go is NOT compacting, and is 2x slower than Julia, so kind of supports my theory.
> **[Erlang Garbage Collector](https://news.ycombinator.com/item?id=15816788)**
>
> 87 points —
> 27 comments —
> signa11 —
> 4:05 PM - 30 Nov 2017
> Go trades the complexity of a compacting GC for forcing you to restart the app regularly if you have a workload with plenty of heap allocation.
Lisp SBCL (2.36x faster) probably compacts, i.e. this is about some alternative, not good enough, _because_ it doesn’t(?):
Parallel garbage collection for SBCL [https://applied-langua.ge/~hayley/swcl-gc.pdf](https://applied-langua.ge/~hayley/swcl-gc.pdf)
> The collector reclaims memory and allows for bump allocation without the collector needing to move objects, using a mark-region heap based on Immix … The parallel garbage collector using one core usually is slower than the copying collector of SBCL, outperforms copying with two cores, and continues to scale with more cores. …
> The collector is not ready to be used in production yet, lacking support for the immobile space of SBCL, and lacking any kind of compaction.
Go moving to compacting/copying GC (but not yet?):
[ ](https://www.youtube.com/watch?v=DpQgJ06ZjGc)
> [@Is the garbage collector non-compacting?](https://discourse.julialang.org/t/is-the-garbage-collector-non-compacting/16335/2):
>
> Afaiu, the GC will never ever become compacting, for reasons of C interoparability. Consider pulling a deepcopy of your data. This is, by the way, not just for fragmentation but will also allocate such that any operation that traverses your structure in deepcopy-order is more cache-friendly.
> [@Julia GC, heap fragmentation, out of memory, push!/append!](https://discourse.julialang.org/t/julia-gc-heap-fragmentation-out-of-memory-push-append/85224):
>
> I saw the juliacon22 video from Christine Flood about Julia GC. I’m glad to see that the GC is finally getting some attention and not just the compiler. It’s long overdue. Let’s just address the elephant in the room: The Julia GC has a memory fragmentation problem. There I said it. I brought this possibility up (I wasn’t sure at that time) on this forum a few times over the years (all the way back since julia 0.2) but this was always brushed aside. The standard answer seems to be: Julia GC doe…
Three garbage collectors: Java, Python, and Julia [https://indico.cern.ch/event/1329685/contributions/5597296/attachments/2777423/4840807/hsf-accelerator-pivarski-gc.pdf](https://indico.cern.ch/event/1329685/contributions/5597296/attachments/2777423/4840807/hsf-accelerator-pivarski-gc.pdf)
> physicists are more interested in Julia than, say, Rust or Lua
Python has 3 generations (also not compacting) and Julia “1 bit = 2 generations”.
---
### Author:  [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)
#### Post date: [November 15, 2024, 10:16am UTC](https://discourse.julialang.org/t/julia-position-in-the-debian-benchmark-game-can-be-improved-and-categorization-of-some-julia-there-is-unfair/122280/27 "2024-11-15T10:16:29Z")
> [@Palli](#):
>
> Java allow heap compaction. Does it make it faster? I can’t actually see how it can be faster, if it moves memory around (that’s an overhead), which is not possible in Julia (since it must be able to call C?), but why was that ever done historically at least? If not for performance reasons?
The big advantage of heap compaction is in allocation: Smallish allocations can be done with a bump allocator, which is very fast. This is because all new allocs come out of an entirely contiguous region of free memory. Hence, heap allocation in java is about as fast as stack allocation in C.
> [@Palli](#):
>
> I’m unclear on how this can work at all without changing all pointers to the still-used regions
This is indeed how this works in java. The GC selects a region with few live objects, and then evacuates the region. For this, it needs to relocate all the live objects to a new one, and rewrite all object references that point to one of the evacuated objects.
Now, with FFI, you have a problem: If some C code has pointers to such an object, then the GC cannot find these pointers, and hence cannot evacuate the region. In other words, the region needs to be pinned, and the user needs to make sure to do the right incantations for this, on pain of hard to debug crashes.
This is a royal pain. JNI / JNA are famous for how painful they are to use. At my dayjob, we recently got hosed by [https://bugs.openjdk.org/browse/JDK-8276094](https://bugs.openjdk.org/browse/JDK-8276094) – before java22, the default G1 garbage collector sucked at the region pinning thing. Hence, your program will randomly crash with OOM if there are too many JNI critical sections that happen to coincide with moments of GC pressure.
What I’m saying is: Compacting GC is better for performance, but makes FFI a pain.
Julia has chosen relatively painless FFI over performance of small heap allocations.
Julia can get away with that choice, because it makes an effort to avoid too many small heap allocs – through language design, API patterns, and escape analysis (JVM hotspot-C2 escape analysis is a joke. Graal got better, though!).
---
### Author:  [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [November 15, 2024, 10:19am UTC](https://discourse.julialang.org/t/julia-position-in-the-debian-benchmark-game-can-be-improved-and-categorization-of-some-julia-there-is-unfair/122280/28 "2024-11-15T10:19:57Z")
> [@foobar\_lv2](#):
>
> This is indeed how this works in java.
You mean without indirection? Ok, you can change pointers, in theory, or in practice seemingly, on the heap (but not for Julia? because of its semantics? You can call C from Java, so how do they manage that?), but you also need to change in the call stack?! ~~And even in registers?~~ It seems like an almost impossible task, ~~or ruling out some register allocation optimization~~? In Julia GC is triggered by allocations, i.e. in the middle of a function. I suppose it can be triggered there in Java too, but likely needs to happen after returning from a call in Java, between calls/stack frames?
> [@foobar\_lv2](#):
>
> before java22, the default G1 garbage collector sucked at the region pinning thing.
Is that about opting out of compacting? I was thinking the same thing, maybe a split heap is worthwhile, one part with compacting, probably for smaller objects, and another non-compacting, for larger objects. Moving is O(n) so likely you don’t want to do that too much for large objects and/or too often. I guess you rely on things settling down, and after a while they don’t move much.
> [@foobar\_lv2](#):
>
> Hence, heap allocation in java is about as fast as stack allocation in C.
But it’s only allocations that get faster, not deallocations, unless done in reverse order. So it seems to me Libc.malloc and Libc.free have sort of same/symmetric performance, but in Java malloc" is basically no cast (I read at some point almost one assembly instruction) while free is not. You of course never do free directly there, but it’s still done by the GC, and the total work is more, even if moved once. So you make up for it in cache effects? Is fragmentation really that bad?
---
### Author:  [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [November 27, 2024, 5:30pm UTC](https://discourse.julialang.org/t/julia-position-in-the-debian-benchmark-game-can-be-improved-and-categorization-of-some-julia-there-is-unfair/122280/29 "2024-11-27T17:30:53Z")
> [@igouy](#):
>
> Look at [the charts page](https://benchmarksgame-team.pages.debian.net/benchmarksgame/box-plot-summary-charts.html). Do you see Julia in those charts or not?
I see Julia, and notice it actually dropped to 7th from 4th place.
> **[Charts showing the fastest programs by language (Benchmarks Game)](https://web.archive.org/web/20240324022322/https://benchmarksgame-team.pages.debian.net/benchmarksgame/box-plot-summary-charts.html)**
>
> Charts showing benchmark program performance grouped by implementation language.
Is it about Julia programs being categorized differently, and only the Julia programs from the general category now in the graph?
It could also be that other languages just got faster since this is a relative rank, but Julia was before within 2x of fastest, now just over 3x (on average), because of outlier mentions it seems, but also the lower outlier seems far of from 1x reading of the logarithmic graph.
If this is ab out categorization (not e.g. change of rank with 1.11) then we should work on getting our best programs possible (back) into the general category.
[It’s intriguing to compare the rank with Fortran. It is missing one program, still has a legal rank. Most often we are close, faster, or slower, usually within 2x, only in few cases much slower or faster.]
---
### Author:  [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [December 12, 2024, 9:55pm UTC](https://discourse.julialang.org/t/julia-position-in-the-debian-benchmark-game-can-be-improved-and-categorization-of-some-julia-there-is-unfair/122280/30 "2024-12-12T21:55:46Z")
> [@foobar\_lv2](#):
>
> Smallish allocations can be done with a bump allocator, which is very fast.
See my experiment with Immix GC:
> [@Improved allocation design, with 4-byte pointers, and sometimes 5-byte in effect](https://discourse.julialang.org/t/improved-allocation-design-with-4-byte-pointers-and-sometimes-5-byte-in-effect/123229/9):
>
> @Oscar_Smith, @GeorgeGkountouras Immix GC (./julia) is confidently 33% faster than Julia 1.11 (with its default GC), for three threads, which is the optimal number for at least than program with (at least) Immix, on this worst-case outlier from Debian Benchmark Game, though only 11% faster than 1.10. Julia 1.11 (default GC) is 17% slower than that Julia 1.10 (default GC), i.e. Immix (1.12.0-DEV.1745) is 33% faster than 1.11. Since the benchmark game uses four threads (all the cores), and I beli…
Such 33% faster (37% faster based on mean) would make Julia climb up to 4.7436 sec. (4.46 sec. if their benchmark uses mean, calculated from 37%) on that benchmark (and improve Julia’s overall rank likely), e.g. climb past C#, and Erlang which is actually faster, and approach Java at 2.62 sec. still 81% slower than it, and 4.9x slower than fastest (C++). We can and should tune it to be able to exploit threads better (now 3 threads, not 4, is optimal).
[Previous page](https://discourse.julialang.org/t/julia-position-in-the-debian-benchmark-game-can-be-improved-and-categorization-of-some-julia-there-is-unfair/122280.md?page=1)