# Bend: a new GPU-native language

**URL:** <https://discourse.julialang.org/t/bend-a-new-gpu-native-language/114440>\
**Category:** Offtopic\
**Tags:** gpu\
**Created:** [May 18, 2024, 10:28pm UTC](https://discourse.julialang.org/t/bend-a-new-gpu-native-language/114440 "2024-05-18T22:28:05Z")\
**Posts on this page:** 1\
**Showing post:** 16

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [May 19, 2024, 3:24pm UTC](https://discourse.julialang.org/t/bend-a-new-gpu-native-language/114440/16 "2024-05-19T15:24:16Z")

</div>

> [@MilesCranmer](#):
>
> It’s not a limited set of constructs that are already easy to write kernels for ([FoldsCUDA.jl](https://juliahub.com/ui/Packages/General/FoldsCUDA)) or a CUDA wrapper ([CUDA.jl](https://juliahub.com/ui/Packages/General/CUDA)), but rather a full GPU-native language that looks like regular code, and it _currently_ generates correct PTX. (and as the saying goes, the last 20% takes 80% of the time)

Ehh, marketing. Read the actual docs, not the marketing material. Bend takes lineage from those kind of DAG-builder approaches but mixes some of Clojure’s pieces in:

> **[Language: Concurrency and Parallelism](https://clojure-doc.org/articles/language/concurrency_and_parallelism/)**
>
> This guide covers:

Where all data structures are immutable and recursive

> Finally, let’s get straight to the fun part: how do we implement parallel algorithms with Bend? Just kidding. Before we get there, let’s talk about loops. You might have noticed we have avoided them so far. That wasn’t by accident. There is an important aspect on which Bend diverges from Python, and aligns with Haskell: **variables are immutable**. Not “by default”. They just **are**. For example, in Bend, we’re not allowed to write:

> <https://github.com/HigherOrderCO/bend/blob/main/GUIDE.md#the-dreaded-immutability>

> Which is immutable. If that sounds annoying, that’s because **it is**. Don’t let anyone tell you otherwise. We are aware of that, and we have many ideas on how to improve this, making Bend feel even more Python-like. For now, we have to live with it. But, wait… if variables are immutable… how do we even do loops?

The solution is of course, transducers. Wait, I mean bend and fold.

> Fold: consuming recursive datatypes  
> Bend: generating recursive datatypes

Bend and fold are just transducers. Transducers.jl comes from the same place.

> Transducers.jl provides composable algorithms on “sequence” of inputs. They are called _[transducers](https://clojure.org/reference/transducers)_ , first introduced in Clojure language by Rich Hickey.

> **[GitHub - JuliaFolds/Transducers.jl: Efficient transducers for Julia](https://github.com/JuliaFolds/Transducers.jl)**
>
> Efficient transducers for Julia. Contribute to JuliaFolds/Transducers.jl development by creating an account on GitHub.

JuliaFolds then built constructs on top of those because most people find writing transducer-based programs quite hard to do:

> **[GitHub - JuliaFolds/Folds.jl: A unified interface for sequential, threaded,...](https://github.com/JuliaFolds/Folds.jl)**
>
> A unified interface for sequential, threaded, and distributed fold - JuliaFolds/Folds.jl

So then it becomes a skeleton-based parallel framework on top of a transducer-based immutable programming layer. Which again I mentioned before as one of the research directions that was ongoing (that I would like to revive). One of the things required to make this more performant in Julia (and thus the barriers for now) include optimizations for immutability and improved escape analysis, which are some major projects right now with the new Memory type of v1.11 and to re-building of constructs on that piece.

So… “rather a full GPU-native language that looks like regular code”, if your regular code only consists of immutable recursive data structures manipulated without loops but instead by transducers then yes, its a full language that looks like regular code! I happen to use loops from time to time, and arrays, and heaps, and some other non-recursive structures. So to me, this is not “regular code”

This of course was the same major adoption barrier to Clojure.

> [@MilesCranmer](#):
>
> Maybe the missing piece here: bend _works_.

No it doesn’t. Parallelism works when it speeds things up. It currently doesn’t speed up code.

But there’s also many projects like this. DawnCC was a C compiler that attempted to automate parallel constructs:

[https://homepages.dcc.ufmg.br/~fernando/publications/papers\_pt/Breno16Tools.pdf](https://homepages.dcc.ufmg.br/~fernando/publications/papers_pt/Breno16Tools.pdf)

Bones was another such project:

> **[GitHub - tue-es/bones: Research compiler based on algorithmic skeletons](https://github.com/tue-es/bones)**
>
> Research compiler based on algorithmic skeletons. Contribute to tue-es/bones development by creating an account on GitHub.

It did well on codes that were amenable to a loop analysis but had difficulties speeding things up that were more general. These types of parallelism models have been called “Skeleton-based programming models”. Other languages to look at in this space include:

> **[SkePU 2: Flexible and Type-Safe Skeleton Programming for Heterogeneous...](https://link.springer.com/article/10.1007/s10766-017-0490-5)**
>
> In this article we present SkePU 2, the next generation of the SkePU C++ skeleton programming framework for heterogeneous parallel systems. We critically examine the design and limitations of the SkePU 1 programming interface. We present a new,...

[https://skelcl.github.io/](https://skelcl.github.io/)

[https://muesli.uni-muenster.de](https://muesli.uni-muenster.de)

[https://parallelme.github.io/](https://parallelme.github.io/)

> **[Valente\_2022.pdf](https://run.unl.pt/bitstream/10362/155512/1/Valente_2022.pdf)**
>
> 5.72 MB

There’s a project from NIST that I’m missing too…

Most are skeleton-based because that’s usually a good way to get something working (i.e. fast enough to do some example well) before taking off to be a whole thing.

Probably the thing I would point to that worked the most in this space so far is hiCUDA, which if you read their papers you’ll see very similar language:

> We have designed hiCUDA,  
> a high-level directive-based language for CUDA programming. It allows programmers to perform these tedious tasks  
> in a simpler manner, and directly to the sequential code.

Its promise was that you could write something pretty close to normal C++ but it would automatically construct the parallel code. The main difference from Bend is that it required you control memory allocations so that you could enforce locality, as this was pretty necessary for the “compiler-guessed CUDA” to get close to the handwritten optimal CUDA. But getting the data handling right is hard, so it was an interesting thing to just opt-out of that part and only focus the automation on the code under the assumption the user will locate the data properly.

> [@MilesCranmer](#):
>
> I’d love to be mistaken of course, but last I checked I still need to write CUDA kernels for any operations that doesn’t fit nicely into broadcasting or reduction. While [CUDA.jl](https://juliahub.com/ui/Packages/General/CUDA) lets you use Julia inside the kernel, it still needs to look like and act like a regular CUDA kernel. And with CUDA kernels there are some things that are hugely complicated to do well, to the point I don’t even bother trying.

There have been many codes to automatically construct DAGs to be alternatively compiled. Those projects have effectively been thrown away because they weren’t actually fast, just like Bend. The pieces that you actually see shared and remaining are pieces with opt-in parallelism because again, anything that was not opt-in was not able to guarantee it wouldn’t slow down normal execution. Removing the opt-in nature is a choice, not a technical hurdle. Just no one has found a good enough solution to the scheduling problem to justify moving to a more automated interface. And clearly Bend hasn’t solved that problem either.

---

_[View the full topic](https://discourse.julialang.org/t/bend-a-new-gpu-native-language/114440)._