Hello everyone! this post will be interesting especially for those who enjoy using 100% of what your computer has to offer, this summer I was part of GSoC and worked to implement multi-streams capabilities for GPUs in Dagger, and we achieved awesome results.
First I want to thank my mentors @fda-tome @jpsamaroo for guiding me all along, helping me every time and being patient, and also my friend @yanzin00 for supporting me and presenting me to GSoC.
Dagger used to launch kernels into a single execution queue, which resulted on low GPU usage and a lot of idle time. Now with multi-streams the GPU can run memory copy and kernels concurrently, maximizing GPU saturation and improving performance massively. We implemented for both CUDA (Nvidia) and ROCm (AMD) backends.
Figure 1: example of use of multi-streams.
It’s very simple to launch a kernel using dagger, all the dependencies will be taken care of automatically
Figure 2: example of code launching a kernel to the GPU
Dagger also offer multi-GPU even with different brands as tested using an AMD and a Nvidia card simultaneously.
Results
We benchmarked across high end computers for both Nvidia and AMD (H200 and MI300A) to evaluate stream scheduling policies and stream count which we are still to give a final answer on which one to use depending on the DAG topology. For now as I’m writing this, I ran the benchmark in my personal computer (RTX 5060 TI 16GB) to generate these images and show the incredible improvement of performance by adding more streams, the results with the high-end computers show similar performance improvements.
The benchmark was run in 4 different DAG shapes: linear, diamond, chainlink and tangled
Figure 3: DAG shapes
Saturation > 1 means that tasks are running concurrently
Figure 4: GPU saturation comparison
Figure 5: Wall time comparison
Next steps
As mentioned before we are still testing different scheduling policies and stream count, and analyzing which scenario is best for each one. We are also applying for presenting a poster about this project at SC26 in Chicago this year.
I really had a great time working in Dagger and I intend to continue improving this project and Julia community in general. If you want to take a further look you can read Dagger documentation and also my development fork. Feel free to suggest new ideas and issues/PRs on GitHub!