Hey everyone,
The last few weeks, I’ve been building my own LLM inference engine because I wanted to try working with Tile-based GPU programming. And since we now have cuTile.jl in the JuliaGPU ecosystem, I did it in Julia.
I’ve been learning a lot of stuff, and I thought it would be a waste to keep it to myself so I wrote a technical-report-style kind of blog post to explain the project, feel free to check it out if you ever wondered how this things works under the hood!
And if you’re extra curious, and already know a fair bit on GPUs and Transformers, I’ve explained the 3 most interesting parts* in more detail in the documentation (KV-Cache, Attention Kernels, Speculative Decoding).
As mentioned at the beginning of the blog post, I’ve mostly made this for myself because I wanted to. I don’t plan on registering it, especially since cuTile.jl is still in beta, but if you somehow have an actual use case for this, don’t hesitate to get in touch!
Blog post: Building an LLM inference engine in Julia to try out cuTile.jl | Antoine Buttier
Repo: GitHub - AntoineBut/TileInfer.jl: A single-GPU LLM inference engine in Julia, built for learning purposes and for fun :D · GitHub
*The metric of interestingness is purely subjective