# SC21: "Comparing Julia to Performance Portable Parallel Programming Models for HPC"

**URL:** <https://discourse.julialang.org/t/sc21-comparing-julia-to-performance-portable-parallel-programming-models-for-hpc/71686>\
**Category:** Julia at Scale\
**Tags:** hpc, llvm, benchmark, scientific-computing, avx\
**Created:** [November 17, 2021, 9:04pm UTC](https://discourse.julialang.org/t/sc21-comparing-julia-to-performance-portable-parallel-programming-models-for-hpc/71686 "2021-11-17T21:04:50Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [November 17, 2021, 9:04pm UTC](https://discourse.julialang.org/t/sc21-comparing-julia-to-performance-portable-parallel-programming-models-for-hpc/71686/1 "2021-11-17T21:04:50Z")

</div>

SCI21 paper:

- [Comparing Julia to Performance Portable Parallel Programming Models for HPC [pdf]](https://scwpub21:conf21%2f%2f@conferences.computer.org/scwpub/pdfs/PMBS2021-vSqRXl4nJSV5KT4jWO5cW/111800a094/111800a094.pdf)
  - Index Terms—Julia, OpenMP, OpenCL, Kokkos, CUDA, HIP, Performance Portability, Programming Models, GPUs

- context: [Presentation • SC21](https://sc21.supercomputing.org/presentation/?id=ws_pmbsf124&sess=sess422) ( 15 November 2021 )
- ( via: Hacker News: [https://news.ycombinator.com/item?id=29257664](https://news.ycombinator.com/item?id=29257664) )

> “VI. CONCLUSION  
> We ported two mini-apps to Julia to show how it is effective  
> at achieving high performance on a range of devices for both  
> memory-bandwidth and compute bound applications.  
> For BabelStream, we observed nearly identical performance  
> to the OpenMP and Kokkos versions on the CPU, with the  
> exception of the Dot kernel on A64FX. As A64FX is a  
> relatively new platform, even LLVM required extra compiler  
> options for optimal performance. We were able to verify that  
> with the latest beta release of Julia, along with a transformation  
> of the reduction kernel to expose the dot product expression,  
> Julia was able to match LLVM’s performance. For GPUs,  
> Julia’s various GPU packages again performed very close  
> (~15%) to first-party frameworks. On AMD platforms, Julia’s  
> AMDGPU.jl package is about 40% slower than the best  
> performing framework, we attribute this to the ROCm stack’s  
> immaturity and AMDGPU.jl’s beta status.  
> For miniBUDE, we get a better view on how well Julia  
> handles floating point optimisations. In general, x86 CPU  
> platforms performed well, although Julia was not able to  
> emit AVX512 on platforms that support it. On AArch64, we  
> observe difficulties for both LLVM and Julia to achieve a  
> high percentage of the theoretical FP32 performance with Julia  
> significantly slower than LLVM results. We believe compiler  
> backends targeting AArch64 have yet to reach the same level  
> of maturity for x86 platforms at the current stage. On GPUs,  
> Julia performed similarly to what OpenCL is getting and it  
> is usually less than 25% difference from the best performing  
> framework. Which is to say, Julia is competitive for compute  
> bound applications on GPUs.  
> Julia largely follows Python’s batteries included motto  
> when it comes to productivity. Many of the GPU packages  
> even handle downloading software dependencies and config-  
> uring the host system for use with Julia’s GPU support. For  
> example, CUDA.jl retrieves the appropriate CUDA SDK from  
> Nvidia on first launch.  
> In addition, because of Julia’s JIT execution model, along  
> with an ergonomic package system, creating a program  
> that supports multiple accelerators from different vendors is  
> straightforward. Traditionally, mixing frameworks that require  
> different host compilers (e.g. nvcc for CUDA or hipcc for  
> HIP) requires special attention to the overall project design  
> to avoid compilation issues; programmers frequently have  
> to resort to compiler-specific workarounds in the codebase  
> and implement fragile and complex build scripts. In fact, the  
> Kokkos framework was designed specifically to abstract this  
> complexity away with highly sophisticated build scripts. Julia  
> was able to avoid all this.  
> Currently, except KA.jl, Julia’s kernel portability is tied to  
> each of the GPU packages. In effect, writing optimised kernels  
> for multiple vendors still requires a manual port. However, as  
> each of the GPU packages share similar capabilities, the effort  
> required is usually limited to basic API call substitutions. We  
> look forward to seeing KA.jl support more platforms under  
> the JuliaGPU umbrella.  
> Thanks to Julia’s approach on reusing large parts of the  
> LLVM project, Julia programs enjoys comparable performance  
> to native C/C++ solutions. And thanks to the concentrated  
> effort from the open-source communities on improving LLVM,  
> Julia gets the unique opportunity to provide best-in-class  
> performance on some of the latest hardware platforms. In  
> general, we find Julia’s language constructs map closely to  
> the underlying LLVM Intermediate Representation under ideal  
> conditions with precisely ascribed types; various conventional  
> optimisation techniques and pitfalls in C/C++ still hold.  
> To this end, Julia offers us a glimpse of what is possible  
> in terms of performance for a managed, dynamically-typed  
> programming language. Given the overall performance-guided  
> design of Julia, the LLVM-backed runtime, and comparable  
> performance results shown here, we think Julia is a strong  
> competitor in achieving a high level of performance portability  
> for HPC use cases.”

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [November 17, 2021, 9:23pm UTC](https://discourse.julialang.org/t/sc21-comparing-julia-to-performance-portable-parallel-programming-models-for-hpc/71686/2 "2021-11-17T21:23:17Z")

</div>

Other links, from the paper:

> Artifacts Available: Source code for JuliaStream.jl is currently in the process of being merged into BabelStream. The Pull Request, along with reviews from members of the Julia community is available at [https://github.com/UoB-HPC/BabelStream/pull/106](https://github.com/UoB-HPC/BabelStream/pull/106). Source code for miniBUDE.jl is now part of the miniBUDE benchmark, available at [GitHub - UoB-HPC/miniBUDE: A BUDE virtual-screening benchmark, in many programming models](https://github.com/UoB-HPC/miniBUDE). We have created scripts to help make the results in this paper reproducible. The source code can be found at [GitHub - UoB-HPC/performance-portability at 2021-benchmarking](https://github.com/UoB-HPC/performance-portability/tree/2021-benchmarking).

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [November 17, 2021, 10:01pm UTC](https://discourse.julialang.org/t/sc21-comparing-julia-to-performance-portable-parallel-programming-models-for-hpc/71686/3 "2021-11-17T22:01:36Z")

</div>

Some comments by @Elrod and myself posted on Slack before this thread existed:

> [Carsten Bauer](https://app.slack.com/team/U881D0W2C) [4 hours ago](https://julialang.slack.com/archives/CBFP2PTTR/p1637170971033400?thread_ts=1637168637.031900&cid=CBFP2PTTR)
> 
> > It is not clear whether Julia currently implements any mechanism for configuring thread affinity policies, tools like numactl may be able to counteract this as a workaround.
> 
> [GitHub - carstenbauer/ThreadPinning.jl: Readily pin Julia threads to CPU processors](https://github.com/carstenbauer/ThreadPinning.jl) but I guess I didn’t create it soon enough However, they could have used [https://juliaperf.github.io/LIKWID.jl/stable/likwid-pin/](https://juliaperf.github.io/LIKWID.jl/stable/likwid-pin/). Anyways, if you use “spread” or (as I call it) “scattered” pinning you will get a curve similar to OpenMP, see e.g. [GitHub - JuliaPerf/BandwidthBenchmark.jl: Measuring memory bandwidth using TheBandwidthBenchmark](https://github.com/JuliaPerf/BandwidthBenchmark.jl#compact-vs-scattered-pinning)

> [Chris Elrod](https://app.slack.com/team/UAUPJLBQX) [4 hours ago](https://julialang.slack.com/archives/CBFP2PTTR/p1637171094033600?thread_ts=1637168637.031900&cid=CBFP2PTTR)
> 
> > For Xeon, we identified  
> > that Julia did not emit AVX512 instructions. On OpenMP,  
> > and by extension, Kokkos, both implementations successfully  
> > emitted AVX512 instructions when compiled using the correct  
> > set of optimisation flags. We were able to confirm that the lack  
> > of AVX512 contributed to the significantly lower performance  
> > (\>30% difference) by replacing -march=skylake-avx512  
> > -mprefer-vector-width=512 with just -march=skylake  
> > on both the OpenMP and Kokkos implementation. With the  
> > non-AVX512 version of OpenMP and Kokkos, Julia showed  
> > nearly identical performance
> 
> FWIW, I always start Julia (on AVX512 platforms) with `-C"native,-prefer-256-bit"` .  
> We could explicitly document this.