Which is the best open model for [Julia] coding? And practical, and still reasonably good

General Usage
1

Continuing the discussion from Anthropic gave me 6 months of Claude Max for my contributions to the Julia ecosystem :slight_smile::

I was just trying to point out an “LLM subscription” is not needed. Local models exist. Qwen was quite good at the time and quantized versions (to extreme binary, or ternary) of it like Bonsai 27 B works fast on even an iPhone:

Kimi K3 is likely best but slowest (not that slow?). Is best on the intriguing ProgramBench benchmark, at 77.8%, just edging out best closed models GPT-5.6 Sol at 77.6% and Claude Fable 5 (max, with fallback) 76.8% and all using KimiCode harness. GLM 5.2 is next best open on that metric (some say not good for Julia, using best harness?), then Kimi K2.7 Code

GLM 5.2 is likely one of the best, I thought impossible to run locally, or the larger DeepSeek V4 [Flash or Pro] but you CAN run the largest Kimi K3 locally, yes slowly:

Note the metric there is s/token, not the usual tokens per second; I’ve seem some claim Kimi K3 is fast (on the CPU!), I think they just read the numbers, and I was careful to point out it slow. It just got stuck in my memory it’s possible at all, I didn’t recall if it was quantized, then could be sped up. Or if using BLAS and/or GPU. It’s memory limited, so if not quantized 16x larger than it strictly needs to be. Could be sped up by that amount roughly then.

2

I think it is really not possible to run Kimi K3 or other really big LLMs locally without proper hardware. Look here: Release AirLLM v3.1.0 — Kimi K3 (2.8T) on a single card · lyogavin/airllm · GitHub, it is incredible, yes, that one can run it with only 4GB of VRAM, though the speed is 292 s/token, which means it isn’t practical for any task, you would wait 10-100 days for a single response :sweat_smile:…maybe there are better implementations but I think it’s kind of impossible that without dedicated hardware one can go anywhere with big local models (for now). The only one which seems maybe worth it is DeepSeek V4 Flash, in the sense, that the necessary hardware is less demanding since it is 281B, but one still needs to have a proper use case to justify not to pay the API which to me it seems at electricity costs (this would an interesting question for me: what is the energy costs and speed in current LLMs with different propor setups).

EDIT: saw that your link says

Is 32.69 seconds per token a verified decode speed? No. It is one author-reported eight-token average that includes first-step prompt work and cold-start effects. It is not a separately measured steady-state decode rate

32 seconds seems better (still way impractical). And doesn’t seem to be verified anyway. I think the same for the other links.

In any case I would bet on Qwen 3.8 27B for the best locally runnable, still not available but will be soon from what I get. Surely though significantly worse than bigger models, at least this seems the case for now, and I don’t think new releases will change this. Though, who knows? Maybe it works fine for some things. Scale seems one dimension which still counts.

3

IMHO, given the price of Deepseek Flash v4 ($0.084 / $0.168 per 1M tokens), you will spend a fraction of what you need considering the hardware, energy, etc. to run locally an acceptable model.

4

I’m afraid @Tortar and @Ronis_BR are right. Unless you’re doing a massive amount of coding and running concurrent requests, it’s probably not economically justifiable to run the latest large models locally on your own hardware. Or at least this is my current understanding.

NVIDIA is offering a decent selection of open-weights models with substantial limits on their free endpoints. Hyper for Charm, among others, offers V4 Flash, and you can get a lot done on their free tier (for workloads suitable for this model).

P.S. BTW, I think we are significantly underestimating Julia in this field. The stack is improving every month. It’s just that the level of entrepreneurship in our community is extremely low, at least in my experience. Moreover, the level of negativity toward this new technology is also very surprising.

5

I tried it inside Claude Code and basically it is a piece of … Not to mentioned that it consumed the credits some 10x faster that even Opus.

Tried also GPT “Sol”. Only slightly better than GLM. And I almost always had to launch Opus to correct the … it did. A huge disappointment, No competition at all vs Claude Code.

My current conclusion is that for more elaborated works there is only Claude Code Opus and above. Sonnet is good sometimes, but the time and limitations makes that Opus was the best choice most of times.

PS I am very jealous of Ronan’s gift (muitos parabéns).

6

Thanks! (Obrigado!) :slight_smile:

7

Kimi k3 seems to do pretty well compared to opus!

8

Locally, I guess those kinds of things are only possible in Germany. :- ) But seriously, how are you running it, @sdanisch? The vLLM requirement seems to be v0.27.0, but v0.26.0 is currently the latest official release.

@Palli, one thing I’d like to add: I have ReactantServer.jl on my waitlist. This looks like a huge deal:

EDIT: Just to be clear, to mitigate any problems. The first sentence is harmless. It’s positive. A kind of a regional “smile” / “joke” / “pun”.

9

Great, you mean Kimi K3 for coding Julia? Good to know [then]. Not just for general programming, or some other specific language? And what harness, if any (I think they are critical, and not all as good) do you use with it?

It’s a good thing AirLLM added:

[2026/07] Kimi K3 (2.8T) support: the largest open-source model runs on a single card in 3.72GB of VRAM, measured end to end on one RTX 6000 Ada. Per-expert streaming loads only the experts a token actually routes to.

Today, we’re open-sourcing Mixture-of-Kittens (MoK), our production MoE training megakernel for NVL72s.

As we have scaled the training and inference of Composer, our agentic coding model, the mixture-of-experts layer has consistently remained the major bottleneck.

It’s twice as fast for e.g. Kimi K2.7 tested there (I suppose would work for K3 too, just wasn’t tested since quite new). “forward throughput” implies to me inference, but I’ve not read yet through all of this and what refers to it or what to training:

DeepEP-based MoK
Tokens / second / GPU 760.9 1,070.2 (1.41x)

Overall, MoK delivered an approximately 41% tokens-per-second speedup over our previous DeepEP-based production setup, allowing us to train our models more efficiently across our GB300 NVL72 infrastructure.

What do you mean waitlist? It’s good to see “large speedups relative to PyTorch (2x to as high as 10x on the same hardware).” but I see it’s for ML however for “non-LLM” models:

Who this is not for

…
LLM serving at scale. vLLM, TGI, TensorRT-LLM, and similar projects are purpose-built for that domain and do it well. This project does not compete in that market.

10

As for AirLLM and Mixture-of-Kittens: I’ll be honest with you, I’m not even planning to touch them. A lot of this is new to me. For example, I initially tried to run GLM 5.2, and it took me about five days to realize it’s probably not possible. The most success I had was with K 2.7, where the best I could achieve was about 8 t/s. However, at 70k tokens, the model starts producing garbage, and as for now, I don’t know why. To get it running, I had to build vLLM, Triton, Flash-Attention, and the rccl-cxi-plugin (based on aws-ofi-rccl) and write a run script. It wasn’t a huge deal, but it wasn’t easy for me either. While I know a bit about Conda, I know almost nothing about Python. On top of that, I’m on a distributed filesystem, so loading these models feels like loading a game from a tape. Imagine how that affects every single iteration.

You know, with a 10x speedup, I’m busy shorting NVIDIA.

Well, as far as my current understanding goes, it is possible to run LLMs as well:

12

Good to know, I suppose it’s the best open weight model, because of its enormous size, so I would also want to know what is second best or by size class. And how slow Kimi K3 needs to be:

For Kimi K3 from sqlite.ai, a token generated every other second (vs “AirLLM running K3 at ~5 minutes per token”) and 10.65 tokens per second for Kimi-Linear model:

recommends 64 GB plus a fast internal SSD. WASTE’s efficiency documentation reports 0.45 to 0.62 tokens per second on a 64 GB M5 Pro. WASTE uses 3-bit residual vector quantization for experts, keeps more sensitive shared weights at 4 or 8 bits, and uses router lookahead to begin reads before the next layer needs them. At 4K context, K3’s compressed KV cache is about 0.21 GB.

.. The documentation also measures Kimi-Linear 48B at 10.65 tokens per second in a 19 GB container with a 1.28 GB minimum. K3 remains the main and best-tested target. .. Conversion takes about 4.7 hours with three workers and needs another 1.42 TB of temporary space.

Another option slightly slower (is it meant for lesser hardware?): GitHub - gavamedia/deltafin: Run full Kimi K3 on a single device. And an OpenAI-compatible API server for local chat and coding agents. · GitHub

This one seemed interesting: Maple-Preview, an AI that runs on iPhones and boasts performance equivalent to Bonsai 27B while being 13 times faster, represents another step forward for local AI. - GIGAZINE

13 times faster than Bonsai 27B [quantized Qwen], which also runs on an iPhone. .. Maple-Preview was reportedly able to accurately solve problems from the International Mathematical Olympiad. [at 281.5 tokens/s on Macbook Pro (M5 Pro)] Maple-Preview consumes only 7.69GB of memory even when handling 131,000 tokens.

I do have only 2 GB VRAM not sure I’m even using the GPU and currently only 32 GB of my 128 GB regular RAM installed, i.e. only one DIMM, would have more channels bandwidth otherwise; 16 cores (hyperthreaded I guess), I changed default from --threads 16 shown in the (modified) llama.cpp github to half the threads, and got more speed, I was happy before:

$ ./build/bin/llama-completion   -m models/maple-preview-TQ2_0-head-Q4_K.gguf   --threads 8   --temp 1.0   --top-p 0.95   --jinja   --conversation
..
0.32.477.396 I common_perf_print: prompt eval time =     268,39 ms /    24 tokens (   11,18 ms per token,    89,42 tokens per second)
0.32.477.404 I common_perf_print:        eval time =   18496,73 ms /   428 runs   (   43,22 ms per token,    23,14 tokens per second)

I tested it with my standard question “What is the Julia language, and can you show me example code?” [first] at:

Neither that model nor MiniMax claim to support coding, maybe they do and I would like to know if good for:

The model fine-tunes Qwen2.5-Coder-14B for Rust-specific programming tasks using a 191K-example synthetic dataset built via multi-model generation and peer-reviewed validation.

I suppose we could do similar for any of [those] models if needed.

13

When I ran out of credits on my Anthropic plan, I tried out Deepseek Flash v4 with pi.dev as the agent. The model was capable enough to port additional ViT backbones for Luximm.jl and also refactor my ViT code to add support “multiscale” outputs that are needed for problems like semantic segmentation, object/keypoint detection, etc. I spent around 50 cents total and it knocked out many other tasks as well. Apparently you can run Flash v4 on two RTX 6000 Pro Blackwell at impressively high token rates, trying to convince my boss to move a few things around so I can have it working on Julia open source stuff full time :sweat_smile:

EDIT: If you are interested in seeing my setup which has extensive sandboxing via bwrap + supply chain hardening, seamless kaimon integration, herdr to manage multiple agents easily, you can see it here: GitHub - csvance/pi-sandbox · GitHub

14

@Palli, to sum up my part of the recent discussions. I think you’ve done a tremendous job with the research and in general popularizing this topic, not only recently but over the course of the last several months. As I indicated in the other threads, I am sustaining my preliminary interest. I believe this is an important topic for the development of Julia (however, I might be very wrong).

At the same time, I think this is potentially a substantial project. For context, I read that training Poolside Laguna S 2.1 (118B parameters) required 4K Nvidia H200 GPUs over a 4-week timeframe. Similarly, Arcee AI Trinity Large (400B parameters) required 2K Nvidia B300 GPUs with a 33-day pretraining window. Furthermore, there is unconfirmed speculation that Moonshot had a computing agreement for around 20K H200 GPUs to train Kimi K3 (2.8T parameters).

While I’m a bit more skeptical than @csvance regarding the capabilities of some of the models mentioned here, I share @Tortar’s concerns about energy and dynamic regulatory environment (I think both are important aspects to monitor), and I certainly share the enthusiasm expressed by @Ronis_BR, @joa-quim, and @sdanisch. (Moça do corpo dourado, do sol de Ipanema, o seu balançado é mais que um poema).

15

I’m not suggesting we train such a model or larger from scratch. I mentioned so-called “fine-tuning” (i.e. postraining) of models, i.e. for Julia (if that is actually needed at all…); such could be done in a day way back, on one GPU. I don’t recall how large such a model it was or exactly when, and I suppose current, larger, models could require longer finetuning. Though that probably needs not be the case, since often you freeze some layers, and only finetune others. Also this thread was mostly about what is the best model now, without too many hacks or finetuning.

16

Flash is on the left. The price has already been raised once.

FIM Completion（Beta） Non-thinking mode only Non-thinking mode only
1M INPUT TOKENS (CACHE HIT) $0.0028 $0.003625
1M INPUT TOKENS (CACHE MISS) $0.14 $0.435
1M OUTPUT TOKENS $0.28 $0.87

(2) We plan to raise the overall pricing for DeepSeek API services in the near future, with a significant increase expected.

So use it while you can… I’m not sure if the announced increase is not yet in effect for those with subscription.

17

My apologies. I was biased by the combined impression of your previous posts, including the other threads. Additionally, your initial post made several references to some of the largest models currently available.

I have not tested every one of them. I was somewhat disappointed with GLM 5.2 and consequently didn’t spend much time with it. Kimi K 2.7 Code is decent, though its extensive thinking process is not always accurate. I occasionally use DeepSeek V4 Flash, which is impressive, particularly for the initial analysis of large logs and documents. Mimo Pro 2.5 also impressed me, although I suspect the CLI contributed significantly to that experience. My primary model is Claude Sonnet 4.5, and I turn to Claude Sonnet 5 and GPT 5.6 Terra from time to time in case of trouble.

Regarding training, as I mentioned in the other thread, I plan to use MaxText to train Qwen3.6-27B for deployment via ReactantServer. I am also researching Trinity Large TrueBase and Base, as I believe there are few recent public examples of such implementations. This is currently on my waitlist, as I have yet to verify the legal, software, and hardware requirements and as many others I have limited time and a few other things on the list that I planned to do.

In general, looking ahead, in the medium to long term, I’m expecting some form of specialization, however, I believe the prospect of running a truly useful model on a mobile device or with only a few GBs of VRAM seems to be a bit too optimistic to me. [P.S. I hope this reply is relevant to the topic. :- )]

18

Modern LLMs can generate code for pretty much any language, not just those languages, which they were trained on, if you provide enough context and proper tools. You can even make your custom DIY language, then feed sources/specs to LLM and it will generate correct code without re-training (I tried this). Small local models, below 100B params, are good only for simple things like auto-completion. For any serious coding and planning you need real big LLM, but not necessarily the top tire one. Cheaper models do good as well, they just take more attempts and more tokens to solve the task. Some providers even give you an estimate of $$/real-world-task for their models. I like Xiaomi MiMo-2.5, which is probably the cheapest one out of capable models.

19

@Palli 1. Reflecting, I wondered if you were referring to distributed computation when you mentioned mobile devices. 2. I recall you sharing a research paper a few months ago regarding distributed training or inference. I haven’t been able to locate the link, but I believe it would be valuable to revisit it, especially in this context. 3. As an addendum to my summary post above, I would like to confirm my preliminary interest again. I currently have nominal access to 256 GPUs for approximately six months, with potential for expansion (TBC - I can’t promise anything). I am not a professional coder / software engineer (Julia, C, q, Redpanda, Oracle ADB, Oracle HeatWave, QuestDB, kdb-X, and BeeGFS). I can dedicate some time to this potential project, however, certainly not on a full-time basis for now.