I just came across the article titled “How OpenAI uses Apache Kafka and Flink for GenAI” and the presentation titled “Building Stream Processing Platform at OpenAI,” delivered at the Current 2025 conference in London. I thought I would post them here for reference in relation to the initial post.
EDIT: I noticed I had linked to the same article twice, so I corrected it.
Finally got a Julia created LLM chatbot made, it can be found at: GitHub - mantzaris/KeemenaLM.jl: Language Models in Julia lang (transformers/GPT/decoders/chat etc) · GitHub
let me know what you think and if you have any ideas for improvements, or want to collaborate
I’m very intrigued by your project(s), commenting more on it below, first questions and more ideas.
You can download your 1 GB model, and I suppose you need the source code to run it, but does it also have all the code and training data (or will download that) to train that model?
I calculate ca. 3 bytes (not bits, let alone 1 bit as possible) per weight max, but it seems you use Float32 so that doesn’t add up. Do you train as such then distribute as Float16? I didn’t see it or bfloat16 in the code, or know of any conversion or quantization. You use GPU with CPU fallback, I suppose you don’t have FP8 or NVFP4 hardware… 4 bits max per weight seems current mainstream, and ternary is possible, even binary,
You did state “expensive” to train, you want to elaborate on that in your case? Time or money?
You can do a 27B model on an iPhone 17 by now (assuming binary quantization), i.e. Qwen-3.6-27B, or after quantization called Bonsai-27 (see table 8), and it’s rather fast, at least on a good laptop.
You could do 4-bit KV cache (“nearly losslessly”, see Table 6.)
Can anyone serve your model on non-Julia code/infrastructure? I suppose you’re not yet thinking of standard formats, so others can’t easily quantize your model (also a bit premature for you or others; post-quant is possible, from FP16, and Bonsai does that, might be even better to take it into account during training).
If you want (more) new (and more radical) ideas:
.. Taking this as inspiration,
we introduce HRM-Text, which replaces standard Transformers with a Hierarchical Recur-
rent Model (HRM) that decouples computation into slow-evolving strategic and fast-evolving
execution layers. To stabilize this deep recurrence for language modeling, we introduce Mag-
icNorm and warmup deep credit assignment. Furthermore, instead of standard raw-text pre-
training, we train exclusively on instruction-response pairs using a task-completion objective
and PrefixLM masking. Serving as an empirical existence proof of efficient pretraining, a 1B-
parameter HRM-Text model trained from scratch on only 40 billion unique tokens and $1,500
budget achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K,
and 56.2% on MATH. Despite utilizing roughly 100-900x fewer training tokens and 96-432x
less estimated compute than standard baselines, HRM-Text performs competitively with 2–7B
parameter open models. ..
Their code is public. Might be no need to reimplement in Julia, but since $1,500 is very cheap ($800 for their smaller model) to train from scratch, it should be very cheap to just fine-tune it.
Another idea is to add a looping capability to your model like here:
Claude Mythos is suspected to be a Recurrent-Depth Transformer (RDT) — also called a Looped Transformer (LT). Rather than stacking hundreds of unique layers, a subset of layers is recycled and run through multiple times per forward pass. Same weights. More loops. Deeper thinking.
Again open code, and models there, no need to reimplement unless you want. You use Adam, they AdamW, optimizer. And HRM-Text uses “Adam-atan2 optimizer”.
I’m very intrigued by this project of your, plus many others, e.g. Home · KeemenaSubwords.jl that seem not to be any of you (direct) dependencies.
I understand this as test cases, 17 pass, those 5 fail, but any idea why (only) those 5, and why you always get 2 spaces in your answers? I’m guessing maybe there’s not a lot of arithmetic in your (chat) training examples. Other reason could be how you tokenize numbers. Fable 5 and Opus 4.8 tokenize 100000 as one token, Opus 4.7 as 7 tokens (according to www.claudetokenizer.com/ might be wrong). I think maybe base-1000 number tokenization might be worthwhile.
It’s known that (mainsteam) tokenization means you need 10x the training data (in gernal/for text, not talking about numbers/digits here), i.e. I’ve seen a paper on a better tokenizer (was done only for English) that helps a lot. Then average token length is longer, and I guess more morphologically meaningful, but note Anthropic changed their tokenizer in Opus 4.8 to go the other direction, shorter average, I suppose for code, lots of punctuation there (also Icelandic got on average way more tokens). It might not matter much in the end, at least for a large model and reasoning. I’vE tRiEd To InDuCe shorter tokens, and with such experiments I get max 3 or even 2 letters per token, and surprisingly the models understand and can respond the same way (tough usually for about a paragraph only, then revert to normal case). This tells me the embeddings, i.e. long words as tokens, like international, or even as two inter-national, may not be too important. For numbers however it makes sense however that base-1000, not longer (or base-10, then only one per digit, not groups of three from the right) tokens might help. That however doesn’t explain your 1 + 1 problem…
WildChat (that; all training examples?) are only mentioned in your Python code (and docs), so what is the Python vs Julia split in the codebase?
Do you support any function calling? FYI: That’s done for at least 3 digit number (like multiplying), not sure with just 1+1, but maybe since Cactus Needle is attention-only i.e. no FFNs (and not for learning any facts, or at least very few, with embeddings and more since there are also other matrices):
Note, intriguingly it’s not decoder-only, rather encoder-decoder:
https://medium.com/@meshuggah22/26m-vs-270m-on-function-calling-i-tested-needle-against-functiongemma-73f31c960c09
.. Taking this as inspiration,
we introduce HRM-Text, which replaces standard Transformers with a Hierarchical Recur-
rent Model (HRM) that decouples computation into slow-evolving strategic and fast-evolving
execution layers. To stabilize this deep recurrence for language modeling, we introduce Mag-
icNorm and warmup deep credit assignment. Furthermore, instead of standard raw-text pre-
training, we train exclusively on instruction-response pairs using a task-completion objective
and PrefixLM masking. Serving as an empirical existence proof of efficient pretraining, a 1B-
parameter HRM-Text model trained from scratch on only 40 billion unique tokens and $1,500
budget achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K,
and 56.2% on MATH. Despite utilizing roughly 100-900x fewer training tokens and 96-432x
less estimated compute than standard baselines, HRM-Text performs competitively with
2–7B parameter open models. ..
I see you use GELU, pretty standard activation function by now (but not for GPT2). You do not use ReLU, but ReLU-squared is interesting and also simple, do people use GELU just to not rock the boat? I suppose you re in general trying to do very standard architecture, to start with?
FYI: I bit off-topic here, but very intriguing/LLM related (I thought of opening a new post for it in off-topic): An Agent Holds the Fort: Three Days of Autonomous Compiler Work | Lab Notes | Rue
The data does not get downloaded but is in the training scripts if you decide to train and replicate. You can just download with the artifacts and run the chatbot fine without the training data.
I did not do any of the quantization tricks but I should have, if there is enough community interest in the project I will go back to it and take on those ideas.
I have worked with the HRM/TRM now for almost a year and have decided to stop after the trials with the traveling saleman on it, it is interesting to see the group apply it to text but I am not convinced that its impressive abilities on tasks like suduku translate to more general problems.
I don’t think we need to limit to pure Julia, could e.g. reuse CUDA/Triton/PyTorch kernels (often super-optimized, and well already existing…) etc. for just catching up. At least for optional parts.
@mantzaris now that we have your LLM code from scratch in Julia, we’ve been discussing improvements, I just hesitate to post a lot of (more) stuff in the package announcement thread itself.
I’m looking for breakthroughs that do not mean we will be playing catch-up, or can avoid implementing some code/model arch that may already be outdated. One possibility is implementing FlashAttention and/or FlexAttention, but maybe defer, in case attention itself is changing a lot. The holy grail I think is memory, as in learning over time, “continual learning”, in some form and one paper related to that (https://www.youtube.com/watch?v=P9uNy71YukQ&t=320s):
However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights.
Regarding attention, maybe implement TANGO and/or WANGO:
A standard Transformer block separates cross-token interaction in self-attention from the nonlinear feed-forward network applied independently at each position. We introduce Token-Aggregated Nonlinear Gating Operators (TANGO), which replaces these two sublayers with a single cross-token gated residual update. Each source token produces a Swish-gated linear unit (SwiGLU) gate vector.
..
The Tango model computes a separate weight for every causally visible source. Windowed Aggregation of Nonlinear Gating Operators (WANGO) retains the same unnormalized pairwise scores within a recent window. For older sources, it uses a positive feature-map weighting rule whose required statistics can be maintained with running sums. Tango is quadratic in sequence length. For fixed window and feature dimensions, the Wango model is linear in sequence length.
..
We compare TANGO and WANGO with Recurrent and Untied Transformer++, full-attention GAU, and FLASH. All models have approximately 44.3M nonembedding parameters and are trained in three matched runs. TANGO, WANGO, and Recurrent Transformer++ apply one shared block four times; the other architectures use four independent blocks.
..
On FineWeb-Edu, the Wango model obtains the lowest mean validation negative log-likelihood (NLL) among models whose computation is linear in sequence length. It also has lower mean NLL than Recurrent Transformer++ at nearly the same analytical forward-pass multiply–accumulate count. The Tango model obtains the lowest mean validation NLL on FineWeb-Edu, Lean, and DeepMind Mathematics, but has the largest analytical forward-pass operation count among the compared architectures
All models use context length 256. Training lasts 84,000 optimizer steps with an effective batch size of 64 examples, corresponding to 5.376 million examples or 32,000 training examples per module–difficulty combination.
This small context might be helpful to you, but note there it applies to DeepMind Mathematics, and I see up to FineWeb-Edu (8,192-token context; 5,722 steps) in figure 2. Figure 5 is also interesting.
“DeepMind Mathematics” is apparently this dataset: GitHub - google-deepmind/mathematics_dataset: This dataset code generates mathematical question and answer pairs, from a range of question types at roughly school-level difficulty. · GitHub
Even if you don’t change your code at all, then it might be good to know of and use such datasets for you.
above paper references this one:
I’ve been very intrigued by looped models, and this seems like an improvement (I added bold):
Importantly, T2MLR does not require pretraining from scratch: retrofitting the recurrent pathway into an existing pretrained 1.7B Transformer and briefly finetuning substantially improves math reasoning, lowering the barrier to practical adoption. These results suggest that effective latent reasoning in Transformers does not require looping over all layers as in previous works, but can instead emerge more strongly from targeted middle-layer recurrence.
Something to consider implementing:
- Monarch Hadamard MLP: replaces the dense FFN with three learnable Walsh-Hadamard-initialized Kronecker (Monarch) factor pairs interleaved with per-channel diagonal scales, fixed permutations, a SiLU nonlinearity, and a rank-8 input-conditioned gate, so each token gets a fully mixed nonlinear transform of its d_model channels at O(d√d) parameters and compute instead of the O(d²) a dense 4x-expansion MLP would cost.
Their attention-only (no FFNs, no knowledge), Simple Attention Networks, and other research is quite intriguing;
But then they added FFNs back in that restricted form, to retain knowledge (gaining speed since better complexity, and smaller models/FFNs, I’m not sure how much knowledge suffers vs full dense FFNs, at least not totally…), where FFNs are not sparse, not dense, but in a block-diagonal format (32x32 blocks I understand, could also be larger). Needle 3 is their ladderiezed variant, and that part also very intriguing:
- Intelligence Laddering: Every layer (2 to 20) is a deployable subnetwork, so one set of weights, 25 to 121 million parameters at 2-bit, shipping as 8-29MB binaries. On a Raspberry Pi 5 it decodes at up to 4k tokens/sec and prefills at up to 10k.
It’s explained here (note they also use engrams):
I believe this explains (this is older background, likely still valid):
https://arxiv.org/pdf/2310.12109
Seems off-topic but related (for graphics?):
They have a file format (no need to support yet) with;
Codebooks, 112 bytes. Twenty-eight floats: the 4-entry, 8-entry and 16-entry Lloyd-Max codebooks for 2-, 3- and 4-bit weights, concatenated so the runtime can index by width. The ternary and binary codebooks are analytic and are not stored.
And we could start with no quantization, then I suppose this applies:
There’s so much happening with Jev now (and even openjev?), and: PrismML — Introducing Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint
I wish someone would make a julia Ai model becuase this is flat bad advice from Fable 5.
OneDnn has enumerators in c++ that start with _ and the suggested fix is terrible if it understood julia treats any _ as write only and this enumerator wrap wouldnt even be accessible from julia if it starts with _ isntead of something like c__ this suggestion came from a different model after trying to wrap _ enums for 3 builds, onednn is the first lbrary ive seen if with underscrore enums
As of Sept 2026, anyone could suggest an implementation of “LLM from scratch all in Julia” with an educational focus? I just thought it would be fun, and Julia syntax provides a much easier-to-understand bedrock than other software combinations.
Are the Medium articles cited earlier still relevant?
You can certainly do it. Here’s a fairly up-to-date open-weight LLM implemented from scratch for inference, and I suppose you can certainly add the training part if you have the time and expertise:
The question is if you can build a community around it for sustained efforts, rather than as a one-off demo project.
Whats it cost to retrain a model? Say a 8b parameter model with open wieghts. Hows that work, you just push new info into the algorithms to switch the open weights?
On my 32GB Ada it took 5 days for the 0.3B model
Can you make a list of the priorities even put it as an issue on the repo?
There is not much in LLM in julia, feel free to join the repo!
Exactly the community organization is ‘hard’ I have been trying to for ages without much success.
Used PyTorch, not a real julia implementation
The Lux.jl implementation was translated from PyTorch but didn’t use PyTorch, to be clear. It’s real Julia code, just not prototyped in Julia.
Good to know
@mantzaris, I think most important to change the optimizer to Aurora to combat “neuron death”.
https://arxiv.org/pdf/2606.27715
You can read about it here:
I previously suggested changing to Muon, but Aurora is state-of-the-art:
nanoGPT speedrun: Aurora achieves a new state-of-the-art, reaching the 3.28 target loss in 3175 steps (previous SoTA: 3225 steps). Pure Aurora without any bells and whistles already ties SoTA at 3225 steps.
.. With 25% fewer params, 100x fewer tokens, and fully open-source internet-only data, we match Qwen3-1.7B on several benchmarks.
..
Four things excite us about this work:Strong empirical gains. We will stress that Aurora 1.1B was trained to be a research artifact → we did not do any tricks with data, architecture, optimizer beyond simply using the Aurora optimizer. Despite this, it was able to achieve 100x data efficiency on several key benchmarks, and we see this as very promising.
Optimizer-architecture co-design. Aurora isn’t a generic optimizer improvement. It’s specifically designed for projections in MLP layers, targeting a pathology that only manifests on tall matrices. In fact, the wider you make your MLP, the more Aurora helps! This level of granularity in optimizer design is rare and, we think, underexplored.
Aurora’s advantage over Muon scales with MLP expansion factor. Delta CE (Aurora − Muon) at step 3000 of 340M-parameter runs across MLP hidden ratios from 0.5× to 16×. More negative = larger Aurora gain.
Dense solutions. Aurora’s row uniformity constraint encourages networks to actually use all their parameters. Models trained with Aurora find denser solutions → the effective parameter count matches the actual parameter count. This suggests there’s significant headroom in existing architectures that’s being left on the table by current optimizers.
Bottom-up optimizer design. ..
I previously suggested Monarch Hadamard MLP, but what it has better complexity, it requires an Engram module. It least in the specialized Cactus Needle 3 I’ve seen it used for. I would say the choice is between those two, and Aurora a better drop-in replacement. The algorithms are short (what I saw in the tweet, I suppose in the paper too, and not more there).
AI tells me: “The absolute simplest, highest-ROI architectural change is to transition your standard FFN to a SwiGLU layer.” and I believe that, and it said “It’s a prerequisite for Aurora”, while later “No, SwiGLU is not strictly mandatory for the Aurora optimizer, but it is highly recommended.” It even has an “Implementation Template in Julia”:
[I’m dismayed Discourse dropped my reply on posting it. EDIT: It DID appear, but out of order at 14d ago. I realized my mistake, I had by accident edited an old post, now it’s recovered. Thanks to @mbauman! and I copied correct reply above, and before realizing what happened wrote below.]
Old reply here (after the original “disappeared”):
I recommend implementing state-of the-art Aurora optimizer (rather than Muon or other I suggested before), to deal with the “dead neuron” problem:
reaches 100x data efficiency over comparable open-source models at 1.1B scale, and is a drop-in replacement with only ~6% overhead.
Code:
I quoted a lot more from that tweet and the paper seems intriguing allowing wider models, and tiny bit from my AI chat that has a Julia implementation:
SwiGLU (or such similar gating) is a likely next recommendation (but seemingly not a “requirement” for Aurora, though strongly recommended also by AI).
I’m very impressed with the research from Tilde, like Aurora I just posted but way more (see their blog), and also their 96 kernels:
E.g. intriguing: Kernel explorer | Popcorn and Kernel explorer | Popcorn and Kernel explorer | Popcorn
I think eventually you will want to add MoE, maybe this way from September:
https://www.researchgate.net/publication/414122432_ACE_Adapter_Consolidation_across_Experts_for_Parameter-Efficient_Fine-Tuning_of_MoE_LLMs
or from last year:
I’ve also suggested FlashAttention before, but Tilde has newer Wall Attention:
The Final Kernel
The fused gradient yields a 12% wall-clock reduction at T=8192 ..
Our Triton implementation is roughly 2x the cost of FlashAttention-2 in isolation. In practice, this overhead is amortized because attention FLOPs are a small fraction of end-to-end training cost (dominated by MLP and embedding layers), so the wall-clock impact on full training is small.
I think we should lean on some non-Julia code like kernels and e.g. Aurora (thought the code is rather small if you want to port it), since the optimizer at least only runs during training. Then we could do pure Julia if we want to for inference… though that is strictly not needed either.
Likely a breakthrough from this month (maybe implement such diffusion LLM, it builds on transformer backbone):
and this, at least cool video:
Decoder-decoder architecture introduced in YOCO in 2024 was new to me (I believe made popular by DeepSeek V4 though there called Causal Encoder-Decoder (CED), so I might be wrong), not to be confused with dual-decoder Transformer from 2020. In Universal YOCO (YOCO-U) from April it’s been very improved (see e.g. Table 1):
https://arxiv.org/pdf/2604.01220
.. We present Universal YOCO (YOCO-U), which combines the YOCO decoder-decoder architecture with recursive computation to achieve a synergistic effect greater than either alone ..
Empirical results confirm that YOCO-U remains highly competitive in general and long-context benchmarks, demonstrating that the integration of efficient-attention architectures and recursive computation is a promising direction for scalable LLMs
Figure 3: Accuracy comparison on 11 math benchmarks. YOCO-U consistently outperforms the YOCO baseline across all tasks, achieving a significant boost in average accuracy.
Prefilling Throughput In the prefilling stage, while Transformer requires O(N^2) time to encode all tokens in parallel, YOCO only requires O(N) time in Self-Decoder. As shown in Figure 7a, although YOCO-U brings additional depth, it still improves the prefilling throughput 10× than vanilla Transformer and 20× than RINS from the complexity advantage. Even at relatively short context such as 16K, YOCO-U still has 1.15× acceleration than non-recursive Transformer. The linear complexity behavior guaranties that the cost of prefilling remains negligible within the overall inference pipeline
See also newer YOCO-S:
Causal Encoder-Decoder (CED) was introduced in DeepSeek-V4.1-Flash (not V4) in September (and I’ve yet to investigate how it’s similar to decoder-decoder/YOLO, despite dissimilar name);
https://arxiv.org/pdf/2609.19969
With its Causal Encoder-Decoder (CED) architecture, the model activates 16B parameters
per token during decode but only 8B parameters during prefill, substantially improving cost
efficiency for agentic workloads.
I learned of both CED and decoder-decoder/YOLO here (but not the newer YOLO variants):
While hybrid architectures such as Samba and the decoder-decoder architecture, YOCO, have shown promising performance gains over Transformers .. In this paper, we introduce the Gated Memory Unit (GMU), a simple yet effective mechanism for efficient memory sharing across layers. We apply it to create SambaY, a decoder-hybrid-decoder architecture that incorporates GMUs.
We introduce a decoder-decoder architecture, YOCO, for large language models, which only caches key-value pairs once.
See also YOCO-S:
Scaling from 1 to 128 agents raises the mean final test-pass rate from 19.31% to 28.78%, an approximately 49% relative improvement. Larger organizations reach comparable test-pass rates earlier. On pandoc, scaling from 1 to 1,024 agents raises the final test-pass rate from 33.89% to 55.06%.
Such BitNet optimizations can likely wait (text from its paper) microsoft/bitnet-embedding-0.6b · Hugging Face
Meanwhile, BITEMBED substantially improves inference efficiency. On CPU with 8 threads, BITEMBED improves token throughput from 364.36 to 830.50 tokens/s on Qwen3-0.6B and from 1181.28 to 2055.47 tokens/s on Gemma3-270M, achieving approximately 2× higher CPU token throughput on both backbones. These results show that the practical utility of BITEMBED in computationally limited scenarios.
