I was just trying to point out an “LLM subscription” is not needed. Local models exist. Qwen was quite good at the time and quantized versions (to extreme binary, or ternary) of it like Bonsai 27 B works fast on even an iPhone:
GLM 5.2 is likely one of the best, I though impossible to run locally, or the larger DeepSeek V4 [Flash or Pro] but you CAN run the largest Kimi K3 locally, yes slowly:
Note the metric there is s/token, not the usual tokens per second; I’ve seem some claim Kimi K3 is fast (on the CPU!), I think they just read the numbers, and I was careful to point out it slow. It just got stuck in my memory it’s possible at all, I didn’t recall if it was quantized, then could be sped up. Or if using BLAS and/or GPU. It’s memory limited, so if not quantized 16x larger than it strictly needs to be. Could be sped up by that amount roughly then.
I think it is really not possible to run Kimi K3 or other really big LLMs locally without proper hardware. Look here: Release AirLLM v3.1.0 — Kimi K3 (2.8T) on a single card · lyogavin/airllm · GitHub, it is incredible, yes, that one can run it with only 4GB of VRAM, though the speed is 292 s/token, which means it isn’t practical for any task, you would wait 10-100 days for a single response …maybe there are better implementations but I think it’s kind of impossible that without dedicated hardware one can go anywhere with big local models (for now). The only one which seems maybe worth it is DeepSeek V4 Flash, in the sense, that the necessary hardware is less demanding since it is 281B, but one still needs to have a proper use case to justify not to pay the API which to me it seems at electricity costs (this would an interesting question for me: what is the energy costs and speed in current LLMs with different propor setups).
EDIT: saw that your link says
Is 32.69 seconds per token a verified decode speed? No. It is one author-reported eight-token average that includes first-step prompt work and cold-start effects. It is not a separately measured steady-state decode rate
32 seconds seems better (still way impractical). And doesn’t seem to be verified anyway. I think the same for the other links.
In any case I would bet on Qwen 3.8 27B for the best locally runnable, still not available but will be soon from what I get. Surely though significantly worse than bigger models, at least this seems the case for now, and I don’t think new releases will change this. Though, who knows? Maybe it works fine for some things. Scale seems one dimension which still counts.
IMHO, given the price of Deepseek Flash v4 ($0.084 / $0.168 per 1M tokens), you will spend a fraction of what you need considering the hardware, energy, etc. to run locally an acceptable model.
:
…maybe there are better implementations but I think it’s kind of impossible that without dedicated hardware one can go anywhere with big local models (for now). The only one which seems maybe worth it is DeepSeek V4 Flash, in the sense, that the necessary hardware is less demanding since it is 281B, but one still needs to have a proper use case to justify not to pay the API which to me it seems at electricity costs (this would an interesting question for me: what is the energy costs and speed in current LLMs with different propor setups).