# Community Interest Check: LLMs from Scratch in Pure Julia

**URL:** <https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796>\
**Category:** Offtopic\
**Tags:** package\
**Created:** [October 26, 2024, 7:10pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796 "2024-10-26T19:10:38Z")\
**Posts on this page:** 20\
**Page:** 4

<div class="post-metadata">

**Author:** ![anon28281882](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon28281882](https://discourse.julialang.org/u/anon28281882)\
**Post date:** [September 24, 2025, 6:38pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/62 "2025-09-24T18:38:53Z")

</div>

> [@mantzaris](#):
>
> ## Project Interest
> 
> 1. Developing pure Julia implementations for LLM training (not just inference)
> 2. Building the necessary distributed training infrastructure
> 3. Creating efficient Julia-native attention mechanisms and optimizers
> 4. Leveraging Julia’s strengths

I just came across the article titled “[How OpenAI uses Apache Kafka and Flink for GenAI](https://www.kai-waehner.de/blog/2025/06/09/how-openai-uses-apache-kafka-and-flink-for-genai/)” and the presentation titled “[Building Stream Processing Platform at OpenAI](https://current.confluent.io/post-conference-videos-2025/building-stream-processing-platform-at-openai-lnd25),” delivered at the Current 2025 conference in London. I thought I would post them here for reference in relation to the initial post.

EDIT: I noticed I had linked to the same article twice, so I corrected it.

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [July 5, 2026, 8:39pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/63 "2026-07-05T20:39:27Z")

</div>

Finally got a Julia created LLM chatbot made, it can be found at: [GitHub - mantzaris/KeemenaLM.jl: Language Models in Julia lang (transformers/GPT/decoders/chat etc) · GitHub](http://github.com/mantzaris/KeemenaLM.jl)

let me know what you think and if you have any ideas for improvements, or want to collaborate

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [July 18, 2026, 6:42pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/64 "2026-07-18T18:42:46Z")

</div>

I’m very intrigued by your project(s), commenting more on it below, first questions and more ideas.

You can download your 1 GB model, and I suppose you need the source code to run it, but does it also have all the code and training data (or will download that) to train that model?

I calculate ca. 3 bytes (not bits, let alone 1 bit as possible) per weight max, but it seems you use Float32 so that doesn’t add up. Do you train as such then distribute as Float16? I didn’t see it or bfloat16 in the code, or know of any conversion or quantization. You use GPU with CPU fallback, I suppose you don’t have FP8 or NVFP4 hardware… 4 bits max per weight seems current mainstream, and ternary is possible, even binary,

You did state “expensive” to train, you want to elaborate on that in your case? Time or money?

You can do a 27B model on an iPhone 17 by now (assuming binary quantization), i.e. Qwen-3.6-27B, or after quantization called Bonsai-27 (see table 8), and it’s rather fast, at least on a good laptop.

You could do 4-bit KV cache (“nearly losslessly”, see Table 6.)

> **[bonsai-27b-whitepaper.pdf](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-27b-whitepaper.pdf)**

> **[PrismML — Announcing Bonsai 27B: The First 27B-Class Model to Run on a Phone](https://prismml.com/news/bonsai-27b)**

Can anyone serve your model on non-Julia code/infrastructure? I suppose you’re not yet thinking of standard formats, so others can’t easily quantize your model (also a bit premature for you or others; post-quant is possible, from FP16, and Bonsai does that, might be even better to take it into account during training).

If you want (more) new (and more radical) ideas:

> **[sapientinc/HRM-Text-1B · Hugging Face](https://huggingface.co/sapientinc/HRM-Text-1B)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

# HRM-Text: Efficient Pretraining Beyond Scaling

> .. Taking this as inspiration,  
> we introduce HRM-Text, which replaces standard Transformers with a Hierarchical Recur-  
> rent Model (HRM) that decouples computation into slow-evolving strategic and fast-evolving  
> execution layers. To stabilize this deep recurrence for language modeling, we introduce _Mag-_  
> _icNorm_ and warmup deep credit assignment. Furthermore, instead of standard raw-text pre-  
> training, we train exclusively on instruction-response pairs using a task-completion objective  
> and PrefixLM masking. Serving as an empirical existence proof of efficient pretraining, a 1B-  
> parameter HRM-Text model trained from scratch on only 40 billion unique tokens and $1,500  
> budget achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K,  
> and 56.2% on MATH. Despite utilizing roughly 100-900x fewer training tokens and 96-432x  
> less estimated compute than standard baselines, HRM-Text performs competitively with 2–7B  
> parameter open models. ..

Their code is public. Might be no need to reimplement in Julia, but since $1,500 is very cheap ($800 for their smaller model) to train from scratch, it should be very cheap to just fine-tune it.

Another idea is to add a looping capability to your model like here:

> **[GitHub - kyegomez/OpenMythos: A theoretical reconstruction of the Claude Mythos...](https://github.com/kyegomez/OpenMythos)**
>
> A theoretical reconstruction of the Claude Mythos architecture, built from first principles using the available research literature.

> Claude Mythos is suspected to be a **Recurrent-Depth Transformer (RDT)** — also called a Looped Transformer (LT). Rather than stacking hundreds of unique layers, a subset of layers is recycled and run through multiple times per forward pass. Same weights. More loops. Deeper thinking.

Again open code, and models there, no need to reimplement unless you want. You use Adam, they AdamW, optimizer. And HRM-Text uses “Adam-atan2 optimizer”.

I’m very intrigued by this project of your, plus many others, e.g. [Home · KeemenaSubwords.jl](https://mantzaris.github.io/KeemenaSubwords.jl/dev/) that seem not to be any of you (direct) dependencies.

> <https://github.com/mantzaris/KeemenaLM.jl/blob/a4c84179db45f8cff0daa27a01ab0d0958ef7743/notes/tiny_chatbot_v9_broad_336m_current_state.md#L141-L151>

I understand this as test cases, 17 pass, those 5 fail, but any idea why (only) those 5, and why you always get 2 spaces in your answers? I’m guessing maybe there’s not a lot of arithmetic in your (chat) training examples. Other reason could be how you tokenize numbers. Fable 5 and Opus 4.8 tokenize 100000 as one token, Opus 4.7 as 7 tokens (according to [www.claudetokenizer.com/](http://www.claudetokenizer.com/) might be wrong). I think maybe base-1000 number tokenization might be worthwhile.

It’s known that (mainsteam) tokenization means you need 10x the training data (in gernal/for text, not talking about numbers/digits here), i.e. I’ve seen a paper on a better tokenizer (was done only for English) that helps a lot. Then average token length is longer, and I guess more morphologically meaningful, but note Anthropic changed their tokenizer in Opus 4.8 to go the other direction, shorter average, I suppose for code, lots of punctuation there (also Icelandic got on average way more tokens). It might not matter much in the end, at least for a large model and reasoning. I’vE tRiEd To InDuCe shorter tokens, and with such experiments I get max 3 or even 2 letters per token, and surprisingly the models understand and can respond the same way (tough usually for about a paragraph only, then revert to normal case). This tells me the embeddings, i.e. long words as tokens, like international, or even as two inter-national, may not be too important. For numbers however it makes sense however that base-1000, not longer (or base-10, then only one per digit, not groups of three from the right) tokens might help. That however doesn’t explain your 1 + 1 problem…

WildChat (that; all training examples?) are only mentioned in your Python code (and docs), so what is the Python vs Julia split in the codebase?

> <https://github.com/mantzaris/KeemenaLM.jl/blob/a4c84179db45f8cff0daa27a01ab0d0958ef7743/tools/prepare_tiny_chatbot_real_chat_corpus.py#L32-L32>

Do you support any function calling? FYI: That’s done for at least 3 digit number (like multiplying), not sure with just 1+1, but maybe since Cactus Needle is attention-only i.e. no FFNs (and not for learning any facts, or at least very few, with embeddings and more since there are also other matrices):

> **[What Is a 26M Parameter Function Calling Model? Cactus Needle Explained](https://www.mindstudio.ai/blog/what-is-cactus-needle-26m-parameter-function-calling-model)**
>
> Cactus Needle is a 26 million parameter model built purely for function calling. Learn how its attention-only architecture beats models 20x larger on tool use.

Note, intriguingly it’s not decoder-only, rather encoder-decoder:  
[https://medium.com/@meshuggah22/26m-vs-270m-on-function-calling-i-tested-needle-against-functiongemma-73f31c960c09](https://medium.com/@meshuggah22/26m-vs-270m-on-function-calling-i-tested-needle-against-functiongemma-73f31c960c09)

> **[sapientinc/HRM-Text-1B · Hugging Face](https://huggingface.co/sapientinc/HRM-Text-1B)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

# HRM-Text: Efficient Pretraining Beyond Scaling

> .. Taking this as inspiration,  
> we introduce HRM-Text, which replaces standard Transformers with a Hierarchical Recur-  
> rent Model (HRM) that decouples computation into slow-evolving strategic and fast-evolving  
> execution layers. To stabilize this deep recurrence for language modeling, we introduce _Mag-_  
> _icNorm_ and warmup deep credit assignment. Furthermore, instead of standard raw-text pre-  
> training, we train exclusively on instruction-response pairs using a task-completion objective  
> and PrefixLM masking. Serving as an empirical existence proof of efficient pretraining, a 1B-  
> parameter HRM-Text model trained from scratch on only 40 billion unique tokens and $1,500  
> budget achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K,  
> and 56.2% on MATH. Despite utilizing roughly 100-900x fewer training tokens and 96-432x  
> less estimated compute than standard baselines, HRM-Text performs competitively with  
> 2–7B parameter open models. ..

I see you use GELU, pretty standard activation function by now (but not for GPT2). You do not use ReLU, but ReLU-squared is interesting and also simple, do people use GELU just to not rock the boat? I suppose you re in general trying to do very standard architecture, to start with?

FYI: I bit off-topic here, but very intriguing/LLM related (I thought of opening a new post for it in off-topic): [An Agent Holds the Fort: Three Days of Autonomous Compiler Work | Lab Notes | Rue](https://rue-lang.dev/blog/an-agent-holds-the-fort/)

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [July 20, 2026, 12:32am UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/65 "2026-07-20T00:32:47Z")

</div>

The data does not get downloaded but is in the training scripts if you decide to train and replicate. You can just download with the artifacts and run the chatbot fine without the training data.

I did not do any of the quantization tricks but I should have, if there is enough community interest in the project I will go back to it and take on those ideas.

I have worked with the HRM/TRM now for almost a year and have decided to stop after the trials with the traveling saleman on it, it is interesting to see the group apply it to text but I am not convinced that its impressive abilities on tasks like suduku translate to more general problems.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [September 1, 2026, 12:36pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/66 "2026-09-01T12:36:22Z")

</div>

> [@Proof of concept LLM chatbot built with Julia: KeemenaLM.jl](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/44):
>
> I think it would be interesting to explore what a Julia-native stack for model development and inference could look like in practice.

I don’t think we need to limit to pure Julia, could e.g. reuse CUDA/Triton/PyTorch kernels (often super-optimized, and well already existing…) etc. for just catching up. At least for optional parts.

> [@Proof of concept LLM chatbot built with Julia: KeemenaLM.jl](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/52):
>
> It would be interesting to see where [KeemenaLM.jl](https://juliaregistries.github.io/General/packages/redirect_to_repo/KeemenaLM) goes next

@mantzaris now that we have your LLM code from scratch in Julia, we’ve been discussing improvements, I just hesitate to post a lot of (more) stuff in the package announcement thread itself.

I’m looking for breakthroughs that do not mean we will be playing catch-up, or can avoid implementing some code/model arch that may already be outdated. One possibility is implementing FlashAttention and/or FlexAttention, but maybe defer, in case attention itself is changing a lot. The holy grail I think is memory, as in learning over time, “continual learning”, in some form and one paper related to that ([https://www.youtube.com/watch?v=P9uNy71YukQ&t=320s](https://www.youtube.com/watch?v=P9uNy71YukQ&t=320s)):

> **[End-to-End Test-Time Training for Long Context](https://arxiv.org/abs/2512.23675)**
>
> We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues...

> However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights.

> **[The Deep View: Conversations | Transcript: #57 - Why AI's next era may not...](https://tdv.transistor.fm/episodes/57-why-ais-next-era-may-not-belong-to-llms-zuzanna-stamirowska/transcript)**
>
> What comes after large language models?
> 
> In this episode of The Deep View Conversations, we talked with Zuzanna Stamirowska, CEO of Pathway, to explore why her team believes today’s dominant AI arc...

> **[When Does Continual Learning Require Learning](https://arxiv.org/abs/2607.07847v1)**
>
> As large language models (LLMs) become increasingly capable, the next question is how can we enable models to continually learn? Today, the field largely frames this as a problem of context management and mitigating forgetting. We argue this framing...

Regarding attention, maybe implement TANGO and/or WANGO:

> **[TANGO: Token-Aggregated Nonlinear Gating Operators for Natural and Formal...](https://arxiv.org/abs/2608.22117)**
>
> A standard Transformer block separates cross-token interaction in self-attention from a nonlinear feed-forward network applied independently at each position. We introduce the TANGO model (Token-Aggregated Nonlinear Gating Operators), which replaces...

> A standard Transformer block separates cross-token interaction in self-attention from the nonlinear feed-forward network applied independently at each position. We introduce Token-Aggregated Nonlinear Gating Operators (TANGO), which replaces these two sublayers with a single cross-token gated residual update. Each source token produces a Swish-gated linear unit (SwiGLU) gate vector.  
> ..  
> The Tango model computes a separate weight for every causally visible source. Windowed Aggregation of Nonlinear Gating Operators (WANGO) retains the same unnormalized pairwise scores within a recent window. For older sources, it uses a positive feature-map weighting rule whose required statistics can be maintained with running sums. Tango is quadratic in sequence length. For fixed window and feature dimensions, the Wango model is linear in sequence length.  
> ..  
> We compare TANGO and WANGO with Recurrent and Untied Transformer++, full-attention GAU, and FLASH. All models have approximately 44.3M nonembedding parameters and are trained in three matched runs. TANGO, WANGO, and Recurrent Transformer++ apply one shared block four times; the other architectures use four independent blocks.  
> ..  
> On FineWeb-Edu, the Wango model obtains the lowest mean validation negative log-likelihood (NLL) among models whose computation is linear in sequence length. It also has lower mean NLL than Recurrent Transformer++ at nearly the same analytical forward-pass multiply–accumulate count. The Tango model obtains the lowest mean validation NLL on FineWeb-Edu, Lean, and DeepMind Mathematics, but has the largest analytical forward-pass operation count among the compared architectures

> All models use context length 256. Training lasts 84,000 optimizer steps with an effective batch size of 64 examples, corresponding to 5.376 million examples or 32,000 training examples per module–difficulty combination.

This small context might be helpful to you, but note there it applies to DeepMind Mathematics, and I see up to FineWeb-Edu (8,192-token context; 5,722 steps) in figure 2. Figure 5 is also interesting.

“DeepMind Mathematics” is apparently this dataset: [GitHub - google-deepmind/mathematics\_dataset: This dataset code generates mathematical question and answer pairs, from a range of question types at roughly school-level difficulty. · GitHub](https://github.com/google-deepmind/mathematics_dataset)

Even if you don’t change your code at all, then it might be good to know of and use such datasets for you.

> **[Full-bandwidth transformer](https://arxiv.org/abs/2608.08888)**
>
> Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal access to the past, but the vertical feedback channel between decoding...

above paper references this one:

I’ve been very intrigued by looped models, and this seems like an improvement (I added bold):

> **[T^2MLR: Transformer with Temporal Middle-Layer Recurrence](https://arxiv.org/abs/2607.15178)**
>
> Transformer reasoning is limited by autoregressive decoding, which repeat edly compresses rich hidden computation through token space and makes it difficult for intermediate reasoning states to persist across time. We in troduce Transformers with...

> Importantly, T2MLR **does not require pretraining from scratch** : retrofitting the recurrent pathway into an existing pretrained 1.7B Transformer **and briefly finetuning substantially improves math reasoning, lowering the barrier to practical adoption**. These results suggest that effective latent reasoning in Transformers **does not require looping over all layers as in previous works** , but can instead emerge more strongly from targeted middle-layer recurrence.

> **[DeepLoop: Depth Scaling for Looped Transformers](https://arxiv.org/abs/2607.13491)**
>
> Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters. This reuse changes the residual-scaling problem: in an untied...

> **[Recirculation: Improve an LLM Without Retraining It](https://mail.bycloud.ai/p/recirculation-improve-an-llm-without-retraining-it)**
>
> plus more about SPADE, skill issue, and Q-learning with World Models

> **[Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive...](https://arxiv.org/abs/2507.10524)**
>
> Scaling language models unlocks impressive capabilities, but the accompanying computational and memory demands make both training and deployment expensive. Existing efficiency efforts typically target either parameter sharing or adaptive computation,...

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [September 19, 2026, 1:34pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/67 "2026-09-19T13:34:27Z")

</div>

Something to consider implementing:

> **[Show HN: Cactus Needle 3: 8-29MB automation models can match DeepSeek V4 Flash](https://news.ycombinator.com/item?id=49748553)**
>
> Hey HN, Henry from Cactus here.\<p\>We submitted Needle 2 here a few weeks ago, and the feedback in the discussion thread was incredibly valuable, thanks! Thanks to all that feedback, we’ve been able to move quickly to release Needle 3 and I'd love to hear what you think again.\<p\>The key features:\<p\>1) Automation (tool calls & structured JSON output): Needle still doesn't chat by...

> 1. Monarch Hadamard MLP: replaces the dense FFN with three learnable Walsh-Hadamard-initialized Kronecker (Monarch) factor pairs interleaved with per-channel diagonal scales, fixed permutations, a SiLU nonlinearity, and a rank-8 input-conditioned gate, so each token gets a fully mixed nonlinear transform of its d\_model channels at O(d√d) parameters and compute instead of the O(d²) a dense 4x-expansion MLP would cost.

Their attention-only (no FFNs, no knowledge), Simple Attention Networks, and other research is quite intriguing;

> **[A Controlled Study of Attention-Only Transformers](https://arxiv.org/abs/2607.18363)**
>
> Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once. We pretrain attention-only decoder transformers (Simple...

But then they added FFNs back in that restricted form, to retain knowledge (gaining speed since better complexity, and smaller models/FFNs, I’m not sure how much knowledge suffers vs full dense FFNs, at least not totally…), where FFNs are not sparse, not dense, but in a block-diagonal format (32x32 blocks I understand, could also be larger). Needle 3 is their ladderiezed variant, and that part also very intriguing:

> 1. Intelligence Laddering: Every layer (2 to 20) is a deployable subnetwork, so one set of weights, 25 to 121 million parameters at 2-bit, shipping as 8-29MB binaries. On a Raspberry Pi 5 it decodes at up to 4k tokens/sec and prefills at up to 10k.

It’s explained here (note they also use engrams):

> **[The Hadamard MLP for Channel Mixing for Almost No Parameters](https://cactuscompute.com/blog/hadamard-mlp)**
>
> Needle replaces the transformer's feed-forward layer with three Kronecker-factored mixes that start as the Walsh-Hadamard transform and learn from there. 25.6K parameters per layer instead of 4.7M, and a fifth of the compute per token.

I believe this explains (this is older background, likely still valid):

> **[Monarch Matrices(M2) instead of Transformers?](https://mlops.substack.com/p/monarch-matricesm2-instead-of-transformers)**
>
> Congress Wants Tech Companies to Pay Up for AI Training Data

[https://arxiv.org/pdf/2310.12109](https://arxiv.org/pdf/2310.12109)

Seems off-topic but related (for graphics?):

> **[CVPR Poster MoST: Efficient Monarch Sparse Tuning for 3D Representation Learning](https://cvpr.thecvf.com/virtual/2025/poster/34500)**

They have a file format (no need to support yet) with;

> **Codebooks, 112 bytes.** Twenty-eight floats: the 4-entry, 8-entry and 16-entry Lloyd-Max codebooks for 2-, 3- and 4-bit weights, concatenated so the runtime can index by width. The ternary and binary codebooks are analytic and are not stored.

And we could start with no quantization, then I suppose this applies:

> **[QAM-W: Joint 2D Codebook Quantization for LLM Weights via Hadamard Rotation...](https://arxiv.org/abs/2605.26339)**
>
> Scalar post-training quantizers discard pairwise coordinate structure within weight rows. We introduce QAM-W (Quadrature Amplitude Modulation for Weights), a codec that recovers this structure: each row is L2-normalized, block-Hadamard rotated,...

There’s so much happening with Jev now (and even openjev?), and: [PrismML — Introducing Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint](https://prismml.com/news/bonsai-2-27b)

---

<div class="post-metadata">

**Author:** ![obsidianjulua](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/obsidianjulua/32/221495_2.png) [@obsidianjulua](https://discourse.julialang.org/u/obsidianjulua)\
**Post date:** [September 21, 2026, 12:13am UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/68 "2026-09-21T00:13:58Z")

</div>

I wish someone would make a julia Ai model becuase this is flat bad advice from Fable 5.

 ![2026-09-20-190955_hyprshot](https://global.discourse-cdn.com/julialang/original/3X/9/b/9b40006473ae1138d00fca254e4e15a199b771f9.png)

OneDnn has enumerators in c++ that start with \_ and the suggested fix is terrible if it understood julia treats any \_ as write only and this enumerator wrap wouldnt even be accessible from julia if it starts with \_ isntead of something like c\_\_ this suggestion came from a different model after trying to wrap \_ enums for 3 builds, onednn is the first lbrary ive seen if with underscrore enums

---

<div class="post-metadata">

**Author:** ![sylvaticus](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaticus/32/203883_2.png) [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Post date:** [September 21, 2026, 8:54am UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/69 "2026-09-21T08:54:54Z")

</div>

As of Sept 2026, anyone could suggest an implementation of “LLM from scratch all in Julia” with an educational focus? I just thought it would be fun, and Julia syntax provides a much easier-to-understand bedrock than other software combinations.  
Are the Medium articles cited earlier still relevant?

---

<div class="post-metadata">

**Author:** ![greatpet](https://avatars.discourse-cdn.com/v4/letter/g/e495f1/32.png) [@greatpet](https://discourse.julialang.org/u/greatpet)\
**Post date:** [September 21, 2026, 11:37am UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/70 "2026-09-21T11:37:51Z")

</div>

You can certainly do it. Here’s a fairly up-to-date open-weight LLM implemented from scratch for inference, and I suppose you can certainly add the training part if you have the time and expertise:

> **[Qwen3 Implementation from Scratch | Lux.jl Docs](https://lux.csail.mit.edu/stable/tutorials/advanced/4_Qwen3)**
>
> Documentation for LuxDL Repositories

The question is if you can build a community around it for sustained efforts, rather than as a one-off demo project.

---

<div class="post-metadata">

**Author:** ![obsidianjulua](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/obsidianjulua/32/221495_2.png) [@obsidianjulua](https://discourse.julialang.org/u/obsidianjulua)\
**Post date:** [September 21, 2026, 11:50am UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/71 "2026-09-21T11:50:25Z")

</div>

Whats it cost to retrain a model? Say a 8b parameter model with open wieghts. Hows that work, you just push new info into the algorithms to switch the open weights?

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [September 22, 2026, 12:23am UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/72 "2026-09-22T00:23:04Z")

</div>

On my 32GB Ada it took 5 days for the 0.3B model

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [September 22, 2026, 12:24am UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/73 "2026-09-22T00:24:17Z")

</div>

Can you make a list of the priorities even put it as an issue on the repo?

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [September 22, 2026, 12:25am UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/74 "2026-09-22T00:25:09Z")

</div>

> [@sylvaticus](#):
>
> t 2026, anyone could suggest an implementation of “LLM from scratch all in Julia” with an educational focus? I just thought it would be fun, and Julia syntax provides a much easier-to-understand bedrock than other software combinations.  
> Are the Medium article

There is not much in LLM in julia, feel free to join the repo!

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [September 22, 2026, 12:25am UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/75 "2026-09-22T00:25:50Z")

</div>

Exactly the community organization is ‘hard’ I have been trying to for ages without much success.

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [September 22, 2026, 12:26am UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/76 "2026-09-22T00:26:18Z")

</div>

Used PyTorch, not a real julia implementation

---

<div class="post-metadata">

**Author:** ![greatpet](https://avatars.discourse-cdn.com/v4/letter/g/e495f1/32.png) [@greatpet](https://discourse.julialang.org/u/greatpet)\
**Post date:** [September 22, 2026, 7:17am UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/77 "2026-09-22T07:17:17Z")

</div>

The Lux.jl implementation was _translated_ from PyTorch but didn’t _use_ PyTorch, to be clear. It’s real Julia code, just not prototyped in Julia.

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [September 22, 2026, 2:14pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/78 "2026-09-22T14:14:05Z")

</div>

Good to know

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [October 4, 2026, 1:21am UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/79 "2026-10-04T01:21:10Z")

</div>

@mantzaris, I think most important to change the optimizer to Aurora to combat “neuron death”.

[https://arxiv.org/pdf/2606.27715](https://arxiv.org/pdf/2606.27715)

You can read about it here:

> <https://x.com/tilderesearch/status/2052797780595499325>

I previously suggested changing to Muon, but Aurora is state-of-the-art:

> **nanoGPT speedrun:** Aurora achieves a new state-of-the-art, reaching the 3.28 target loss in 3175 steps (previous SoTA: 3225 steps). Pure Aurora without any bells and whistles already ties SoTA at 3225 steps.
> 
> .. With 25% fewer params, 100x fewer tokens, and fully open-source internet-only data, we match Qwen3-1.7B on several benchmarks.  
> ..  
> Four things excite us about this work:
> 
> **Strong empirical gains.** We will stress that Aurora 1.1B was trained to be a research artifact → we did not do any tricks with data, architecture, optimizer beyond simply using the Aurora optimizer. _Despite this, it was able to achieve 100x data efficiency on several key benchmarks, and we see this as very promising._
> 
> **Optimizer-architecture co-design.** Aurora isn’t a generic optimizer improvement. It’s specifically designed for projections in MLP layers, targeting a pathology that only manifests on tall matrices. In fact, the wider you make your MLP, the more Aurora helps! This level of granularity in optimizer design is rare and, we think, underexplored.
> 
> [![Image](https://global.discourse-cdn.com/julialang/original/3X/0/0/003fa55a777f3f975cdc2415138abde516cd6181.jpeg)](https://x.com/tilderesearch/article/2052797780595499325/media/2052794593310302208)
> 
> Aurora’s advantage over Muon scales with MLP expansion factor. Delta CE (Aurora − Muon) at step 3000 of 340M-parameter runs across MLP hidden ratios from 0.5× to 16×. More negative = larger Aurora gain.
> 
> **Dense solutions.** Aurora’s row uniformity constraint encourages networks to actually use all their parameters. Models trained with Aurora find denser solutions → the effective parameter count matches the actual parameter count. This suggests there’s significant headroom in existing architectures that’s being left on the table by current optimizers.
> 
> **Bottom-up optimizer design.**..

I previously suggested Monarch Hadamard MLP, but what it has better complexity, it requires an Engram module. It least in the specialized Cactus Needle 3 I’ve seen it used for. I would say the choice is between those two, and Aurora a better drop-in replacement. The algorithms are short (what I saw in the tweet, I suppose in the paper too, and not more there).

AI tells me: “The absolute **simplest, highest-ROI architectural change** is to transition your standard FFN to a **SwiGLU** layer.” and I believe that, and it said “It’s a prerequisite for Aurora”, while later “ **No, SwiGLU is not strictly mandatory for the Aurora optimizer, but it is highly recommended.** ” It even has an “Implementation Template in Julia”:

> **[Google Search](https://www.google.com/share.google?q=aimode%2FP1qExlDxMUE9ZBIuW)**

[I’m dismayed Discourse ~~dropped~~ my reply on posting it. EDIT: It DID appear, but out of order at 14d ago. I realized my mistake, I had by accident edited an old post, now it’s recovered. Thanks to @mbauman! and I copied correct reply above, and before realizing what happened wrote below.]

Old reply here (after the original “disappeared”):

I recommend implementing state-of the-art Aurora optimizer (rather than Muon or other I suggested before), to deal with the “dead neuron” problem:

> <https://x.com/tilderesearch/status/2052797780595499325>

> reaches 100x data efficiency over comparable open-source models at 1.1B scale, and is a drop-in replacement with only ~6% overhead.
> 
> Code:
> 
> [github.com/tilde-research/aurora-release](https://github.com/tilde-research/aurora-release)

I quoted a lot more from that tweet and the paper seems intriguing allowing wider models, and tiny bit from my AI chat that has a Julia implementation:

> **[Google Search](https://www.google.com/share.google?q=aimode%2FP1qExlDxMUE9ZBIuW)**

SwiGLU (or such similar gating) is a likely next recommendation (but seemingly not a “requirement” for Aurora, though strongly recommended also by AI).

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [October 4, 2026, 7:58pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/80 "2026-10-04T19:58:35Z")

</div>

I’m very impressed with the research from Tilde, like Aurora I just posted but way more (see their blog), and also their 96 kernels:

> **[Popcorn](https://tilde-research.github.io/popcorn/)**
>
> Reference-checked, benchmark-driven kernel dispatch for PyTorch.

E.g. intriguing: [Kernel explorer | Popcorn](https://tilde-research.github.io/popcorn/kernels?s=wall_attn) and [Kernel explorer | Popcorn](https://tilde-research.github.io/popcorn/kernels?s=momoe) and [Kernel explorer | Popcorn](https://tilde-research.github.io/popcorn/kernels?s=rwkv7)

I think eventually you will want to add MoE, maybe this way from September:  
[https://www.researchgate.net/publication/414122432\_ACE\_Adapter\_Consolidation\_across\_Experts\_for\_Parameter-Efficient\_Fine-Tuning\_of\_MoE\_LLMs](https://www.researchgate.net/publication/414122432_ACE_Adapter_Consolidation_across_Experts_for_Parameter-Efficient_Fine-Tuning_of_MoE_LLMs)

or from last year:

> **[GitHub - tilde-research/momoe-release: Memory optimized Mixture of Experts](https://github.com/tilde-research/momoe-release)**
>
> Memory optimized Mixture of Experts

I’ve also suggested FlashAttention before, but Tilde has newer Wall Attention:

> **[Wall Attention: Length Generalization With Diagonal Gates | Tilde](https://blog.tilderesearch.com/blog/wall-attn)**

> ### The Final Kernel
> 
> The fused gradient yields a 12% wall-clock reduction at T=8192 ..
> 
> Our Triton implementation is roughly 2x the cost of FlashAttention-2 in isolation. In practice, this overhead is amortized because attention FLOPs are a small fraction of end-to-end training cost (dominated by MLP and embedding layers), so the wall-clock impact on full training is small.

I think we should lean on some non-Julia code like kernels and e.g. Aurora (thought the code is rather small if you want to port it), since the optimizer at least only runs during training. Then we could do pure Julia if we want to for inference… though that is strictly not needed either.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [October 5, 2026, 2:18am UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/81 "2026-10-05T02:18:55Z")

</div>

Likely a breakthrough from this month (maybe implement such diffusion LLM, it builds on transformer backbone):

> **[Paper page - Hierarchical Continuous Diffusion Language Models](https://huggingface.co/papers/2610.02193)**
>
> Join the discussion on this paper page

and this, at least cool video:

> **[Paper page - World Observer: Joint Actor-Observer Generation for Persistent...](https://huggingface.co/papers/2610.02162)**
>
> Join the discussion on this paper page

Decoder-decoder architecture introduced in YOCO in 2024 was new to me (I believe made popular by DeepSeek V4 though there called Causal Encoder-Decoder (CED), so I might be wrong), not to be confused with dual-decoder Transformer from 2020. In Universal YOCO (YOCO-U) from April it’s been very improved (see e.g. Table 1):  
[https://arxiv.org/pdf/2604.01220](https://arxiv.org/pdf/2604.01220)

> .. We present Universal YOCO (YOCO-U), which combines the YOCO decoder-decoder architecture with recursive computation to achieve a synergistic effect greater than either alone ..  
> Empirical results confirm that YOCO-U remains highly competitive in general and long-context benchmarks, demonstrating that the integration of efficient-attention architectures and recursive computation is a promising direction for scalable LLMs

> Figure 3: Accuracy comparison on 11 math benchmarks. YOCO-U consistently outperforms the YOCO baseline across all tasks, achieving a significant boost in average accuracy.

> **Prefilling** Throughput In the prefilling stage, while Transformer requires O(N^2) time to encode all tokens in parallel, YOCO only requires O(N) time in Self-Decoder. As shown in Figure 7a, although YOCO-U brings additional depth, it still improves the prefilling throughput 10× than vanilla Transformer and 20× than RINS from the complexity advantage. Even at relatively short context such as 16K, YOCO-U still has 1.15× acceleration than non-recursive Transformer. The linear complexity behavior guaranties that the cost of prefilling remains negligible within the overall inference pipeline

See also newer YOCO-S:

> **[You Only Index Once: Cross-Layer Sparse Attention with Shared Routing](https://arxiv.org/abs/2606.06467)**
>
> Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought. Existing sparse attention methods often face a practical...

Causal Encoder-Decoder (CED) was introduced in DeepSeek-V4.1-Flash (not V4) in September (and I’ve yet to investigate how it’s similar to decoder-decoder/YOLO, despite dissimilar name);  
[https://arxiv.org/pdf/2609.19969](https://arxiv.org/pdf/2609.19969)

> With its Causal Encoder-Decoder (CED) architecture, the model activates 16B parameters  
> per token during decode but only 8B parameters during prefill, substantially improving cost  
> efficiency for agentic workloads.

I learned of both CED and decoder-decoder/YOLO here (but not the newer YOLO variants):

[![](https://global.discourse-cdn.com/julialang/original/3X/9/1/910d11db590f1c3bea8e4f491fa41b1ac1734b1e.jpeg "DeepSeek Splits the Transformer in Two. Here’s Why.") ](https://www.youtube.com/watch?v=qkngB1Q46CY)

> **[Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation](https://arxiv.org/abs/2507.06607)**
>
> Recent advances in language modeling have demonstrated the effectiveness of State Space Models (SSMs) for efficient sequence modeling. While hybrid architectures such as Samba and the decoder-decoder architecture, YOCO, have shown promising...

> While hybrid architectures such as Samba and the decoder-decoder architecture, YOCO, have shown promising performance gains over Transformers .. In this paper, we introduce the Gated Memory Unit (GMU), a simple yet effective mechanism for efficient memory sharing across layers. We apply it to create SambaY, a decoder-hybrid-decoder architecture that incorporates GMUs.

> **[You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://papers.nips.cc/paper_files/paper/2024/hash/0df38cd13520747e1e64e5b123a78ef8-Abstract-Conference.html)**

> We introduce a decoder-decoder architecture, YOCO, for large language models, which only caches key-value pairs once.

See also YOCO-S:

> **[You Only Index Once: Cross-Layer Sparse Attention with Shared Routing](https://arxiv.org/abs/2606.06467)**
>
> Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought. Existing sparse attention methods often face a practical...

> **[Agensh: Scaling Organizational Intelligence to 1,024 Agents](https://arxiv.org/abs/2609.26781)**
>
> A multi-agent system can reduce latency on complex tasks by executing work concurrently. Several pioneering harness frameworks support multi-agent systems. However, the scalability of current multi-agent harnesses is often constrained by a central...

> Scaling from 1 to 128 agents raises the mean final test-pass rate from 19.31% to 28.78%, an approximately 49% relative improvement. Larger organizations reach comparable test-pass rates earlier. On pandoc, scaling from 1 to 1,024 agents raises the final test-pass rate from 33.89% to 55.06%.

Such BitNet optimizations can likely wait (text from its paper) [microsoft/bitnet-embedding-0.6b · Hugging Face](https://huggingface.co/microsoft/bitnet-embedding-0.6b)

> Meanwhile, BITEMBED substantially improves inference efficiency. On CPU with 8 threads, BITEMBED improves token throughput from 364.36 to 830.50 tokens/s on Qwen3-0.6B and from 1181.28 to 2055.47 tokens/s on Gemma3-270M, achieving approximately 2× higher CPU token throughput on both backbones. These results show that the practical utility of BITEMBED in computationally limited scenarios.

> **[VibeVoice-ASR-BitNet Technical Report](https://arxiv.org/abs/2607.21075)**
>
> We present VibeVoice-ASR-BitNet, a compressed variant of VibeVoice-ASR optimized for real-time inference on edge CPUs. We apply heterogeneous quantization tailored to the computational characteristics of each stage: the VAE acoustic tokenizer uses...

> **[Advancing AI for Humanity](https://thegenerality.com/agi/)**

[Previous page](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796.md?page=3)