# Sparse matrix handling (in Julia) and Sparse is Enough in Scaling Transformers paper

**URL:** <https://discourse.julialang.org/t/sparse-matrix-handling-in-julia-and-sparse-is-enough-in-scaling-transformers-paper/72455>\
**Category:** Offtopic\
**Created:** [December 2, 2021, 2:53pm UTC](https://discourse.julialang.org/t/sparse-matrix-handling-in-julia-and-sparse-is-enough-in-scaling-transformers-paper/72455 "2021-12-02T14:53:40Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [December 2, 2021, 2:53pm UTC](https://discourse.julialang.org/t/sparse-matrix-handling-in-julia-and-sparse-is-enough-in-scaling-transformers-paper/72455/1 "2021-12-02T14:53:41Z")

</div>

Where does Julia stand regarding sparse matrices for neural networks or otherwise? They have generally relied on dense matrix multiply, until now:

> **[Sparse is Enough in Scaling Transformers](https://arxiv.org/abs/2111.12763)**
>
> Large Transformer models yield impressive results on many tasks, but are expensive to train, or even fine-tune, and so slow at decoding that their use and study becomes out of reach. We address this problem by leveraging sparsity. We study sparse...

> The improvement in complexity holds not just asymptotically but yields over 2.6x speedup in wall-clock hed decoding time already for a model with 800M parameters and 20x improvement for a model with 17B parameters, as shown in Table 1.  
> […]  
> 3 Sparse is Enough  
> We study how to sparsify every part of the Transformer model—otherwise the non-sparse parts  
> dominate decoding time and become a bottleneck.  
> […]  
> While integrating sparse attention layers into a Scaling Transformer, we notice that the architecture  
> of the Transformer decoder block is suboptimal and can be redesigned […] We therefore remove the encoder-decoder attention

This seems like a game-changer, and could tilt neural networks in Julia’s favor.

It seems this new 2021 paper could help:

> **[A Pattern-Based SpGEMM Library for Multi-Core and Many-Core Architectures](https://ieeexplore.ieee.org/document/9459513)**
>
> General sparse matrix-matrix multiplication (SpGEMM) is one of the most important mathematical library routines in a number of applications. In recent years, several efficient SpGEMM algorithms have been proposed, however, most of them are based on...

> In recent years, several efficient SpGEMM algorithms have been proposed, however, most of them are based on the compressed sparse row (CSR) format, and the possible performance gain from exploiting other formats has not been well studied. […] We then propose a pattern-based SpGEMM library, that provides a unified programming interface in the CSR format, analyses the pattern of two input matrices, and automatically determines the best format, algorithm, and parameter for arbitrary matrix pairs. For this purpose, we build an algorithm set that integrates three new designed algorithms with existing popular libraries, and design a hybrid deep learning model called MatNet to quickly identify patterns of input matrices and accurately predict the best solution by using sparse features and density representations. The evaluation shows that this library consistently outperforms the state-of-the-art library.

> **[SpArch: Efficient Architecture for Sparse Matrix Multiplication](https://ieeexplore.ieee.org/document/9065428)**
>
> Generalized Sparse Matrix-Matrix Multiplication (SpGEMM) is a ubiquitous task in various engineering and scientific applications. However, inner product based SpGEMM introduces redundant input fetches for mismatched nonzero operands, while outer...

> On average, SpArch achieves 4x, 19x, 18x, 17x, 1285x speedup and 6x, 164x, 435x, 307x, 62x energy savings over OuterSpace, MKL, cuSPARSE, CUSP, and ARM Armadillo, respectively.

> **[A Systematic Survey of General Sparse Matrix-Matrix Multiplication](https://arxiv.org/abs/2002.11273)**
>
> General Sparse Matrix-Matrix Multiplication (SpGEMM) has attracted much attention from researchers in graph analyzing, scientific computing, and deep learning. Many optimization techniques have been developed for different applications and computing...

See hee: [https://github.com/google/trax/blob/master/trax/examples/Terraformer\_from\_scratch.ipynb](https://github.com/google/trax/blob/master/trax/examples/Terraformer_from_scratch.ipynb)

> This colab contains all relevant code for the paper “Sparse is Enough in Scaling Transformers”. We depend on the Trax library

[![](https://global.discourse-cdn.com/julialang/original/3X/5/2/52d63e2d41a4d5203210a287ea611272af22abbd.jpeg "Sparse is Enough in Scaling Transformers (aka Terraformer) | ML Research Paper Explained") ](https://www.youtube.com/watch?v=hgSGHusDx7M)