# How to profile Julia MPI code?

**URL:** <https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136>\
**Category:** Julia at Scale\
**Tags:** mpi\
**Created:** [March 14, 2021, 5:01pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136 "2021-03-14T17:01:52Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![PolarizedPoutine](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/polarizedpoutine/32/6067_2.png) [@PolarizedPoutine](https://discourse.julialang.org/u/PolarizedPoutine)\
**Post date:** [March 14, 2021, 5:01pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/1 "2021-03-14T17:01:52Z")

</div>

How do people usually profile Julia MPI code to find bottlenecks and reduce communication overhead?

As an MPI noob I thought there would be some profiler tools out (kinda like `cuprof` and NVIDIA Nsight) there but they seem to be mostly proprietary (e.g. TotalView, Arm MAP) and probably overkill for our use case since we’re not running with thousands of ranks _yet_.

Looks like there may be some open-source language-independent tools like [http://www.hpctoolkit.org/](http://www.hpctoolkit.org/) and [https://github.com/LLNL/mpiP](https://github.com/LLNL/mpiP) which I might try looking into.

EDIT: Right now just trying to profile CPU MPI code, but definitely looking to profile GPU/CUDA-aware MPI code in the near future.

---

<div class="post-metadata">

**Author:** ![viralbshah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viralbshah/32/54_2.png) [@viralbshah](https://discourse.julialang.org/u/viralbshah)\
**Post date:** [March 14, 2021, 5:25pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/2 "2021-03-14T17:25:21Z")

</div>

I would love to see graphs like in this paper by Parry Husbands and Kathy Yelick: [https://upc.lbl.gov/publications/husbands-lu-sc07.pdf](https://upc.lbl.gov/publications/husbands-lu-sc07.pdf) Figures 4 and 5. I imagine one can easily instrument the MPI, added the timers, and then log it all - one could produce various nice visualizations like we do with our profilers. I think they wrote some custom tooling for the charts in that paper.

We could do similar tricks as libblastrampoline to make this all nice and easy for MPI, but @staticfloat says ABI issues in MPI prevent this.

I think Dagger.jl has some of this tooling now. I am sure I have seen pictures of such MPI communication visualizers at conferences that make me feel like there are products that do this and I found this SO thread: [openmpi - Visualizing communication pattern of MPI processes - Stack Overflow](https://stackoverflow.com/questions/23161396/visualizing-communication-pattern-of-mpi-processes)

-viral

---

<div class="post-metadata">

**Author:** ![antoine-levitt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antoine-levitt/32/4008_2.png) [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)\
**Post date:** [March 14, 2021, 6:31pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/3 "2021-03-14T18:31:24Z")

</div>

If you have relatively simple patterns and small amount of ranks you can get decent results just by profiling a single rank. You can slap a TimerOutput on your MPI calls to get a nice output of whether communication is limiting. If you have load balancing issues you can also time the operations and compute eg stddevs. Of course that doesn’t replace a proper parallel profiler for large scale applications, but for smaller setups it’s a good quick and dirty solution.

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [March 15, 2021, 3:55pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/4 "2021-03-15T15:55:51Z")

</div>

Thanks to @lyonsquark, v0.17 of MPI.jl (which I just tagged) should now support MPI profilers which use `LD_PRELOAD` hooks. I believe he has tested it with [Darshan](https://www.mcs.anl.gov/research/projects/darshan/).

I haven’t tried it, but I believe you should also be able to use [NVIDIA Nsight Systems](https://docs.nvidia.com/nsight-systems/UserGuide/index.html) to profile MPI even if you’re not using CUDA: just specify `--trace=mpi` option (you will also need to specify the MPI implementation via the `--mpi-impl` option).

I’d be keen to hear how people get on with various MPI profilers: if you do have problems (or find solutions to problems), please chime in here or open an issue.

---

<div class="post-metadata">

**Author:** ![staticfloat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/staticfloat/32/34_2.png) [@staticfloat](https://discourse.julialang.org/u/staticfloat)\
**Post date:** [March 15, 2021, 4:52pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/5 "2021-03-15T16:52:35Z")

</div>

> We could do similar tricks as libblastrampoline to make this all nice and easy for MPI, but @staticfloat says ABI issues in MPI prevent this.

I haven’t looked into it too deeply, but I think an MPI demuxing library is definitely doable. It wouldn’t be the same as libblastrampoline because the ABI is actually different between the different vendors (as opposed to BLAS where the different vendors tend to differ in only naming and/or ILP64-ness), so you’d need to do a bit of argument translation. I haven’t looked into how you would autodetect the ABI, but I’m sure it’s possible. It’s not high on my TODO list, but I am confident we can do something similar if we really want to.

---

<div class="post-metadata">

**Author:** ![simonbyrne](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/simonbyrne/32/19_2.png) [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)\
**Post date:** [March 15, 2021, 6:02pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/6 "2021-03-15T18:02:27Z")

</div>

> [@staticfloat](#):
>
> I haven’t looked into how you would autodetect the ABI, but I’m sure it’s possible.

This is the logic MPI.jl currently uses: [https://github.com/JuliaParallel/MPI.jl/blob/078723f8a052c7af863e0e70cf6dc3007fdc5d65/src/implementations.jl#L80-L147](https://github.com/JuliaParallel/MPI.jl/blob/078723f8a052c7af863e0e70cf6dc3007fdc5d65/src/implementations.jl#L80-L147)

I think it would make sense limit support to Open MPI, MPICH and Microsoft MPI ABIs, which would cover almost all current MPI implementations (other than \>5 year old MPICH derivatives). The main challenge is how to define handles: they can be either 32-bit integers or pointers.

---

<div class="post-metadata">

**Author:** ![richardreeve](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/richardreeve/32/2257_2.png) [@richardreeve](https://discourse.julialang.org/u/richardreeve)\
**Post date:** [March 15, 2021, 9:43pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/7 "2021-03-15T21:43:03Z")

</div>

I can’t say this is what I did myself, but we got NAG involved in profiling our (CPU) MPI Julia code (for which we are very grateful!), and they used Extrae to profile it, providing some extremely clear results showing where outstanding problems were in great detail. We had to get advice on slack about how to instrument the libraries correctly, but it all worked out relatively(!) easily in the end (and maybe these recent fixes have solved those problems), except that we didn’t seem to be able to go beyond 127 MPI processes for some as-yet-unresolved reason…

---

<div class="post-metadata">

**Author:** ![PolarizedPoutine](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/polarizedpoutine/32/6067_2.png) [@PolarizedPoutine](https://discourse.julialang.org/u/PolarizedPoutine)\
**Post date:** [March 16, 2021, 6:45pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/8 "2021-03-16T18:45:50Z")

</div>

Thanks for the helpful replies everyone! We’ll try @simonbyrne’s suggestion of using NVIDIA Nsight since it might allow us to kill two birds with one stone (GPU profiling and MPI profiling) and post back!

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [August 29, 2021, 5:55pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/9 "2021-08-29T17:55:13Z")

</div>

@PolarizedPoutine Have you had success with using `nsys`? I’m currently playing around with using it for regular non-mpi CPU profiling but so far with limited success.

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [March 29, 2023, 2:13pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/10 "2023-03-29T14:13:41Z")

</div>

Update: Two primitive examples for using NVIDIA Nsight Systems to profile Julia MPI code: [https://github.com/carstenbauer/JuliaHLRS22/tree/main/backup/MPI%20profiling%20(nsys)](https://github.com/carstenbauer/JuliaHLRS22/tree/main/backup/MPI%20profiling%20(nsys))

 ![image](https://global.discourse-cdn.com/julialang/original/3X/4/2/4215a344ee2b481475e5554509412b95623f4967.png)

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [March 29, 2023, 2:19pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/11 "2023-03-29T14:19:09Z")

</div>

And since I’m anyway necroposting, [here](https://github.com/JuliaPerf/ScoreP.jl) is an experimental (but working) attempt to profile Julia MPI code with [Score-P.jl](https://github.com/JuliaPerf/ScoreP.jl).

 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/e/fe8a4a070c04b3becc370a83f71e369b57b7f18f.png)

(Would be great to start similar efforts for [Extrae](https://tools.bsc.es/extrae), [HPCToolkit](https://hpc.llnl.gov/software/development-environment-software/hpc-toolkit), [TAU](https://www.cs.uoregon.edu/research/tau/home.php), etc.)

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [March 29, 2023, 2:22pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/12 "2023-03-29T14:22:10Z")

</div>

Are `get_arguments` and `integration` Julia functions?

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [March 29, 2023, 2:23pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/13 "2023-03-29T14:23:46Z")

</div>

No, unfortunately not. They are manually assigned names of NVTX ranges (see, e.g., [here](https://github.com/carstenbauer/JuliaHLRS22/blob/main/backup/MPI%20profiling%20(nsys)/trapezoidal_int/sendrecv/mpi_trapezoidal.jl#L47)).

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [March 29, 2023, 2:25pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/14 "2023-03-29T14:25:16Z")

</div>

> [@carstenbauer](#):
>
> Would be great to start similar efforts for [Extrae](https://tools.bsc.es/extrae)

Maybe @mofeing would be interested in this? 👀

---

<div class="post-metadata">

**Author:** ![mofeing](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mofeing/32/47619_2.png) [@mofeing](https://discourse.julialang.org/u/mofeing)\
**Post date:** [March 29, 2023, 2:45pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/15 "2023-03-29T14:45:06Z")

</div>

Yes! Actually we have already started some efforts in [Extrae.jl](https://github.com/bsc-quantic/Extrae.jl) and can find us discussing in Slack’s #extrae channel.

We managed to trace some toy examples working with Distributed but we are crashing on some more complex examples, posible due to hacks 😅.

We would thank the help from a Julia internals expert.

---

<div class="post-metadata">

**Author:** ![clasqui](https://avatars.discourse-cdn.com/v4/letter/c/ee59a6/32.png) [@clasqui](https://discourse.julialang.org/u/clasqui)\
**Post date:** [March 29, 2023, 2:48pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/16 "2023-03-29T14:48:39Z")

</div>

Hey! Yes, was going to reply this also. We are focusing on instrumenting the `Distributed` model for the moment, and then we’ll tackle the Threads part.

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [March 29, 2023, 2:55pm UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/17 "2023-03-29T14:55:41Z")

</div>

As for your Figures 4 and 5: Needs some visual fine-tuning but my (drafty) [MPITape.jl](https://github.com/pc2/MPITape.jl) can produce this:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/e/3/e3f7162af653ae8f219201e47d45495609e09d21.png)

(Unfortunately?) It is based on Cassette.jl and might come with an overhead though. Would be great to have an efficient, non-Cassette based tracing API in Julia. But I think some people are working on something in this direction if I’m not mistaken (see #profiling on Slack).

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 3, 2024, 12:16am UTC](https://discourse.julialang.org/t/how-to-profile-julia-mpi-code/57136/18 "2024-10-03T00:16:05Z")

</div>

I’d be curious to hear whether there were any further developments of MPI tracing. Anyone? In particular, there is MPE. Is there any Julia code to process MPE log files?