I just came across the article titled “How OpenAI uses Apache Kafka and Flink for GenAI” and the presentation titled “Building Stream Processing Platform at OpenAI,” delivered at the Current 2025 conference in London. I thought I would post them here for reference in relation to the initial post.
EDIT: I noticed I had linked to the same article twice, so I corrected it.
Finally got a Julia created LLM chatbot made, it can be found at: GitHub - mantzaris/KeemenaLM.jl: Language Models in Julia lang (transformers/GPT/decoders/chat etc) · GitHub
let me know what you think and if you have any ideas for improvements, or want to collaborate
I’m very intrigued by your project(s), commenting more on it below, first questions and more ideas.
You can download your 1 GB model, and I suppose you need the source code to run it, but does it also have all the code and training data (or will download that) to train that model?
I calculate ca. 3 bytes (not bits, let alone 1 bit as possible) per weight max, but it seems you use Float32 so that doesn’t add up. Do you train as such then distribute as Float16? I didn’t see it or bfloat16 in the code, or know of any conversion or quantization. You use GPU with CPU fallback, I suppose you don’t have FP8 or NVFP4 hardware… 4 bits max per weight seems current mainstream, and ternary is possible, even binary,
You did state “expensive” to train, you want to elaborate on that in your case? Time or money?
You can do a 27B model on an iPhone 17 by now (assuming binary quantization), i.e. Qwen-3.6-27B, or after quantization called Bonsai-27 (see table 8), and it’s rather fast, at least on a good laptop.
You could do 4-bit KV cache (“nearly losslessly”, see Table 6.)
Can anyone serve your model on non-Julia code/infrastructure? I suppose you’re not yet thinking of standard formats, so others can’t easily quantize your model (also a bit premature for you or others; post-quant is possible, from FP16, and Bonsai does that, might be even better to take it into account during training).
If you want (more) new (and more radical) ideas:
.. Taking this as inspiration,
we introduce HRM-Text, which replaces standard Transformers with a Hierarchical Recur-
rent Model (HRM) that decouples computation into slow-evolving strategic and fast-evolving
execution layers. To stabilize this deep recurrence for language modeling, we introduce Mag-
icNorm and warmup deep credit assignment. Furthermore, instead of standard raw-text pre-
training, we train exclusively on instruction-response pairs using a task-completion objective
and PrefixLM masking. Serving as an empirical existence proof of efficient pretraining, a 1B-
parameter HRM-Text model trained from scratch on only 40 billion unique tokens and $1,500
budget achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K,
and 56.2% on MATH. Despite utilizing roughly 100-900x fewer training tokens and 96-432x
less estimated compute than standard baselines, HRM-Text performs competitively with 2–7B
parameter open models. ..
Their code is public. Might be no need to reimplement in Julia, but since $1,500 is very cheap ($800 for their smaller model) to train from scratch, it should be very cheap to just fine-tune it.
Another idea is to add a looping capability to your model like here:
Claude Mythos is suspected to be a Recurrent-Depth Transformer (RDT) — also called a Looped Transformer (LT). Rather than stacking hundreds of unique layers, a subset of layers is recycled and run through multiple times per forward pass. Same weights. More loops. Deeper thinking.
Again open code, and models there, no need to reimplement unless you want. You use Adam, they AdamW, optimizer. And HRM-Text uses “Adam-atan2 optimizer”.
I’m very intrigued by this project of your, plus many others, e.g. Home · KeemenaSubwords.jl that seem not to be any of you (direct) dependencies.
I understand this as test cases, 17 pass, those 5 fail, but any idea why (only) those 5, and why you always get 2 spaces in your answers? I’m guessing maybe there’s not a lot of arithmetic in your (chat) training examples. Other reason could be how you tokenize numbers. Fable 5 and Opus 4.8 tokenize 100000 as one token, Opus 4.7 as 7 tokens (according to www.claudetokenizer.com/ might be wrong). I think maybe base-1000 number tokenization might be worthwhile.
It’s known that (mainsteam) tokenization means you need 10x the training data (in gernal/for text, not talking about numbers/digits here), i.e. I’ve seen a paper on a better tokenizer (was done only for English) that helps a lot. Then average token length is longer, and I guess more morphologically meaningful, but note Anthropic changed their tokenizer in Opus 4.8 to go the other direction, shorter average, I suppose for code, lots of punctuation there (also Icelandic got on average way more tokens). It might not matter much in the end, at least for a large model and reasoning. I’vE tRiEd To InDuCe shorter tokens, and with such experiments I get max 3 or even 2 letters per token, and surprisingly the models understand and can respond the same way (tough usually for about a paragraph only, then revert to normal case). This tells me the embeddings, i.e. long words as tokens, like international, or even as two inter-national, may not be too important. For numbers however it makes sense however that base-1000, not longer (or base-10, then only one per digit, not groups of three from the right) tokens might help. That however doesn’t explain your 1 + 1 problem…
WildChat (that; all training examples?) are only mentioned in your Python code (and docs), so what is the Python vs Julia split in the codebase?
Do you support any function calling? FYI: That’s done for at least 3 digit number (like multiplying), not sure with just 1+1, but maybe since Cactus Needle is attention-only i.e. no FFNs (and not for learning any facts, or at least very few, with embeddings and more since there are also other matrices):
Note, intriguingly it’s not decoder-only, rather encoder-decoder:
https://medium.com/@meshuggah22/26m-vs-270m-on-function-calling-i-tested-needle-against-functiongemma-73f31c960c09
.. Taking this as inspiration,
we introduce HRM-Text, which replaces standard Transformers with a Hierarchical Recur-
rent Model (HRM) that decouples computation into slow-evolving strategic and fast-evolving
execution layers. To stabilize this deep recurrence for language modeling, we introduce Mag-
icNorm and warmup deep credit assignment. Furthermore, instead of standard raw-text pre-
training, we train exclusively on instruction-response pairs using a task-completion objective
and PrefixLM masking. Serving as an empirical existence proof of efficient pretraining, a 1B-
parameter HRM-Text model trained from scratch on only 40 billion unique tokens and $1,500
budget achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K,
and 56.2% on MATH. Despite utilizing roughly 100-900x fewer training tokens and 96-432x
less estimated compute than standard baselines, HRM-Text performs competitively with
2–7B parameter open models. ..
I see you use GELU, pretty standard activation function by now (but not for GPT2). You do not use ReLU, but ReLU-squared is interesting and also simple, do people use GELU just to not rock the boat? I suppose you re in general trying to do very standard architecture, to start with?
FYI: I bit off-topic here, but very intriguing/LLM related (I thought of opening a new post for it in off-topic): An Agent Holds the Fort: Three Days of Autonomous Compiler Work | Lab Notes | Rue
The data does not get downloaded but is in the training scripts if you decide to train and replicate. You can just download with the artifacts and run the chatbot fine without the training data.
I did not do any of the quantization tricks but I should have, if there is enough community interest in the project I will go back to it and take on those ideas.
I have worked with the HRM/TRM now for almost a year and have decided to stop after the trials with the traveling saleman on it, it is interesting to see the group apply it to text but I am not convinced that its impressive abilities on tasks like suduku translate to more general problems.
I don’t think we need to limit to pure Julia, could e.g. reuse CUDA/Triton/PyTorch kernels (often super-optimized, and well already existing…) etc. for just catching up. At least for optional parts.
@mantzaris now that we have your LLM code from scratch in Julia, we’ve been discussing improvements, I just hesitate to post a lot of (more) stuff in the package announcement thread itself.
I’m looking for breakthroughs that do not mean we will be playing catch-up, or can avoid implementing some code/model arch that may already be outdated. One possibility is implementing FlashAttention and/or FlexAttention, but maybe defer, in case attention itself is changing a lot. The holy grail I think is memory, as in learning over time, “continual learning”, in some form and one paper related to that (https://www.youtube.com/watch?v=P9uNy71YukQ&t=320s):
However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights.
Regarding attention, maybe implement TANGO and/or WANGO:
A standard Transformer block separates cross-token interaction in self-attention from the nonlinear feed-forward network applied independently at each position. We introduce Token-Aggregated Nonlinear Gating Operators (TANGO), which replaces these two sublayers with a single cross-token gated residual update. Each source token produces a Swish-gated linear unit (SwiGLU) gate vector.
..
The Tango model computes a separate weight for every causally visible source. Windowed Aggregation of Nonlinear Gating Operators (WANGO) retains the same unnormalized pairwise scores within a recent window. For older sources, it uses a positive feature-map weighting rule whose required statistics can be maintained with running sums. Tango is quadratic in sequence length. For fixed window and feature dimensions, the Wango model is linear in sequence length.
..
We compare TANGO and WANGO with Recurrent and Untied Transformer++, full-attention GAU, and FLASH. All models have approximately 44.3M nonembedding parameters and are trained in three matched runs. TANGO, WANGO, and Recurrent Transformer++ apply one shared block four times; the other architectures use four independent blocks.
..
On FineWeb-Edu, the Wango model obtains the lowest mean validation negative log-likelihood (NLL) among models whose computation is linear in sequence length. It also has lower mean NLL than Recurrent Transformer++ at nearly the same analytical forward-pass multiply–accumulate count. The Tango model obtains the lowest mean validation NLL on FineWeb-Edu, Lean, and DeepMind Mathematics, but has the largest analytical forward-pass operation count among the compared architectures
All models use context length 256. Training lasts 84,000 optimizer steps with an effective batch size of 64 examples, corresponding to 5.376 million examples or 32,000 training examples per module–difficulty combination.
This small context might be helpful to you, but note there it applies to DeepMind Mathematics, and I see up to FineWeb-Edu (8,192-token context; 5,722 steps) in figure 2. Figure 5 is also interesting.
“DeepMind Mathematics” is apparently this dataset: GitHub - google-deepmind/mathematics_dataset: This dataset code generates mathematical question and answer pairs, from a range of question types at roughly school-level difficulty. · GitHub
Even if you don’t change your code at all, then it might be good to know of and use such datasets for you.
above paper references this one:
I’ve been very intrigued by looped models, and this seems like an improvement (I added bold):
Importantly, T2MLR does not require pretraining from scratch: retrofitting the recurrent pathway into an existing pretrained 1.7B Transformer and briefly finetuning substantially improves math reasoning, lowering the barrier to practical adoption. These results suggest that effective latent reasoning in Transformers does not require looping over all layers as in previous works, but can instead emerge more strongly from targeted middle-layer recurrence.