As for AirLLM and Mixture-of-Kittens: I’ll be honest with you, I’m not even planning to touch them. A lot of this is new to me. For example, I initially tried to run GLM 5.2, and it took me about five days to realize it’s probably not possible. The most success I had was with K 2.7, where the best I could achieve was about 8 t/s. However, at 70k tokens, the model starts producing garbage, and as for now, I don’t know why. To get it running, I had to build vLLM, Triton, Flash-Attention, and the rccl-cxi-plugin (based on aws-ofi-rccl) and write a run script. It wasn’t a huge deal, but it wasn’t easy for me either. While I know a bit about Conda, I know almost nothing about Python. On top of that, I’m on a distributed filesystem, so loading these models feels like loading a game from a tape. Imagine how that affects every single iteration.
You know, with a 10x speedup, I’m busy shorting NVIDIA.
Well, as far as my current understanding goes, it is possible to run LLMs as well: