Continuing the discussion from Proof of concept LLM chatbot built with Julia: KeemenaLM.jl:
JetSpec reaches 9.64x on MATH-500 and 4.58x on open-ended chat, and these gains carry into real single-stream serving on JetSpec’s own engine with an average of around 1000 TPS throughput on MATH-500 using a single B200 GPU.
That’s up to 9.64x faster inference vs “only” (still great) up to 6.12x faster with the mainstream DFlash (and DDTree also beats it).
@mantzaris I really appreciate the work you done, I just realize how monumentially difficult it is to beat the mainstream LLMs, since you/we’re are so behind on pre-training compute budget. I alternate between pessimism and optimism because of interesting research we could implement. We would always be behind, and while Julia could do all this, all else equal, others can too, and the mindshare is elsewhere. I think we should consider rather fine-tuning an already good model for Julia, i.e. for some local model to be better at coding in Julia. And/or work on inference speedup. DFlash is strictly a fully separate contained component, and yes such a draft model is often bundled with downloads. It’s premature if you want to work on smaller models, since such drafting models are an add-on.
For those that worry about energy use of (inference and/or) pretraining of AI/LLM, it’s currently being done with 3x more memory than needed, meaning “if a model was only trainable on a H100 that costs like $30,000 US, it is now trainable on a RTX490 that’s only like $2,000” because of Sakana’s DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation
i.e The Most Absurd Way To Train LLMs… With 3x Less Memory!? https://www.youtube.com/watch?v=T_GhB7lK2YE&t=309s