# Community Interest Check: LLMs from Scratch in Pure Julia

**URL:** <https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796>\
**Category:** Offtopic\
**Tags:** package\
**Created:** [October 26, 2024, 7:10pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796 "2024-10-26T19:10:38Z")\
**Posts on this page:** 1\
**Showing post:** 66

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [September 1, 2026, 12:36pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/66 "2026-09-01T12:36:22Z")

</div>

> [@Proof of concept LLM chatbot built with Julia: KeemenaLM.jl](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/44):
>
> I think it would be interesting to explore what a Julia-native stack for model development and inference could look like in practice.

I don’t think we need to limit to pure Julia, could e.g. reuse CUDA/Triton/PyTorch kernels (often super-optimized, and well already existing…) etc. for just catching up. At least for optional parts.

> [@Proof of concept LLM chatbot built with Julia: KeemenaLM.jl](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/52):
>
> It would be interesting to see where [KeemenaLM.jl](https://juliaregistries.github.io/General/packages/redirect_to_repo/KeemenaLM) goes next

@mantzaris now that we have your LLM code from scratch in Julia, we’ve been discussing improvements, I just hesitate to post a lot of (more) stuff in the package announcement thread itself.

I’m looking for breakthroughs that do not mean we will be playing catch-up, or can avoid implementing some code/model arch that may already be outdated. One possibility is implementing FlashAttention and/or FlexAttention, but maybe defer, in case attention itself is changing a lot. The holy grail I think is memory, as in learning over time, “continual learning”, in some form and one paper related to that ([https://www.youtube.com/watch?v=P9uNy71YukQ&t=320s](https://www.youtube.com/watch?v=P9uNy71YukQ&t=320s)):

> **[End-to-End Test-Time Training for Long Context](https://arxiv.org/abs/2512.23675)**
>
> We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues...

> However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights.

> **[The Deep View: Conversations | Transcript: #57 - Why AI's next era may not...](https://tdv.transistor.fm/episodes/57-why-ais-next-era-may-not-belong-to-llms-zuzanna-stamirowska/transcript)**
>
> What comes after large language models?
> 
> In this episode of The Deep View Conversations, we talked with Zuzanna Stamirowska, CEO of Pathway, to explore why her team believes today’s dominant AI arc...

> **[When Does Continual Learning Require Learning](https://arxiv.org/abs/2607.07847v1)**
>
> As large language models (LLMs) become increasingly capable, the next question is how can we enable models to continually learn? Today, the field largely frames this as a problem of context management and mitigating forgetting. We argue this framing...

Regarding attention, maybe implement TANGO and/or WANGO:

> **[TANGO: Token-Aggregated Nonlinear Gating Operators for Natural and Formal...](https://arxiv.org/abs/2608.22117)**
>
> A standard Transformer block separates cross-token interaction in self-attention from a nonlinear feed-forward network applied independently at each position. We introduce the TANGO model (Token-Aggregated Nonlinear Gating Operators), which replaces...

> A standard Transformer block separates cross-token interaction in self-attention from the nonlinear feed-forward network applied independently at each position. We introduce Token-Aggregated Nonlinear Gating Operators (TANGO), which replaces these two sublayers with a single cross-token gated residual update. Each source token produces a Swish-gated linear unit (SwiGLU) gate vector.  
> ..  
> The Tango model computes a separate weight for every causally visible source. Windowed Aggregation of Nonlinear Gating Operators (WANGO) retains the same unnormalized pairwise scores within a recent window. For older sources, it uses a positive feature-map weighting rule whose required statistics can be maintained with running sums. Tango is quadratic in sequence length. For fixed window and feature dimensions, the Wango model is linear in sequence length.  
> ..  
> We compare TANGO and WANGO with Recurrent and Untied Transformer++, full-attention GAU, and FLASH. All models have approximately 44.3M nonembedding parameters and are trained in three matched runs. TANGO, WANGO, and Recurrent Transformer++ apply one shared block four times; the other architectures use four independent blocks.  
> ..  
> On FineWeb-Edu, the Wango model obtains the lowest mean validation negative log-likelihood (NLL) among models whose computation is linear in sequence length. It also has lower mean NLL than Recurrent Transformer++ at nearly the same analytical forward-pass multiply–accumulate count. The Tango model obtains the lowest mean validation NLL on FineWeb-Edu, Lean, and DeepMind Mathematics, but has the largest analytical forward-pass operation count among the compared architectures

> All models use context length 256. Training lasts 84,000 optimizer steps with an effective batch size of 64 examples, corresponding to 5.376 million examples or 32,000 training examples per module–difficulty combination.

This small context might be helpful to you, but note there it applies to DeepMind Mathematics, and I see up to FineWeb-Edu (8,192-token context; 5,722 steps) in figure 2. Figure 5 is also interesting.

“DeepMind Mathematics” is apparently this dataset: [GitHub - google-deepmind/mathematics\_dataset: This dataset code generates mathematical question and answer pairs, from a range of question types at roughly school-level difficulty. · GitHub](https://github.com/google-deepmind/mathematics_dataset)

Even if you don’t change your code at all, then it might be good to know of and use such datasets for you.

> **[Full-bandwidth transformer](https://arxiv.org/abs/2608.08888)**
>
> Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal access to the past, but the vertical feedback channel between decoding...

above paper references this one:

I’ve been very intrigued by looped models, and this seems like an improvement (I added bold):

> **[T^2MLR: Transformer with Temporal Middle-Layer Recurrence](https://arxiv.org/abs/2607.15178)**
>
> Transformer reasoning is limited by autoregressive decoding, which repeat edly compresses rich hidden computation through token space and makes it difficult for intermediate reasoning states to persist across time. We in troduce Transformers with...

> Importantly, T2MLR **does not require pretraining from scratch** : retrofitting the recurrent pathway into an existing pretrained 1.7B Transformer **and briefly finetuning substantially improves math reasoning, lowering the barrier to practical adoption**. These results suggest that effective latent reasoning in Transformers **does not require looping over all layers as in previous works** , but can instead emerge more strongly from targeted middle-layer recurrence.

> **[DeepLoop: Depth Scaling for Looped Transformers](https://arxiv.org/abs/2607.13491)**
>
> Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters. This reuse changes the residual-scaling problem: in an untied...

> **[Recirculation: Improve an LLM Without Retraining It](https://mail.bycloud.ai/p/recirculation-improve-an-llm-without-retraining-it)**
>
> plus more about SPADE, skill issue, and Q-learning with World Models

> **[Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive...](https://arxiv.org/abs/2507.10524)**
>
> Scaling language models unlocks impressive capabilities, but the accompanying computational and memory demands make both training and deployment expensive. Existing efficiency efforts typically target either parameter sharing or adaptive computation,...

---

_[View the full topic](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796)._