Diffusion-based drafting LLM breakthroughs, and maybe we should work on fine-tuning for Julia; and inference rather than pretraining?

Offtopic
1

Continuing the discussion from Proof of concept LLM chatbot built with Julia: KeemenaLM.jl:

JetSpec reaches 9.64x on MATH-500 and 4.58x on open-ended chat, and these gains carry into real single-stream serving on JetSpec’s own engine with an average of around 1000 TPS throughput on MATH-500 using a single B200 GPU.

That’s up to 9.64x faster inference vs “only” (still great) up to 6.12x faster with the mainstream DFlash (and DDTree also beats it).

@mantzaris I really appreciate the work you done, I just realize how monumentially difficult it is to beat the mainstream LLMs, since you/we’re are so behind on pre-training compute budget. I alternate between pessimism and optimism because of interesting research we could implement. We would always be behind, and while Julia could do all this, all else equal, others can too, and the mindshare is elsewhere. I think we should consider rather fine-tuning an already good model for Julia, i.e. for some local model to be better at coding in Julia. And/or work on inference speedup. DFlash is strictly a fully separate contained component, and yes such a draft model is often bundled with downloads. It’s premature if you want to work on smaller models, since such drafting models are an add-on.

For those that worry about energy use of (inference and/or) pretraining of AI/LLM, it’s currently being done with 3x more memory than needed, meaning “if a model was only trainable on a H100 that costs like $30,000 US, it is now trainable on a RTX490 that’s only like $2,000” because of Sakana’s DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation

i.e The Most Absurd Way To Train LLMs… With 3x Less Memory!? https://www.youtube.com/watch?v=T_GhB7lK2YE&t=309s

2

Could you summarize a plan? I am not sure practically how fine tuning in general is going to be straightforward if we don’t have onnx up to par and if the model exports are not standardized. Would be great if we could get a group together that is dedicated.

3

I’m no expert on fine-tuning (or middle tuning). I was thinking just get a good recent model like (then fine-tune on Julia, but not with Julia tools):

If not Kimi K3… or some model like ornith-ai/Ornith-1.5-35B-A3B · Hugging Face

I see some prefer Qwen3.6 to 3.8 for some things… while stating 3.8 beats for other things. Qwen-3.8-Max may be best but I don’t find it publicly… https://qwen.ai/blog?id=qwen3.8

10+ Days of Autonomous Coding: Building a Self-Evolving Harness

…

Reproduce a research paper — then improve it

…

Autonomous Chip Design and Closed-Loop Feedback-Driven Optimization

I’m really intrigued by 3.8-Max (since I’m now looking into chip design…). The new DeepSeek harness is really intriguing, and it might be best or Prime Agent (both intriguing, not sure which is better, since also both rather new).

Some old process:

FYI: @cpfiffer other threads on this, maybe further discussion should continue there: