Similar to @pitsianis, I’ve added and tested preliminary AMDGPU support. I haven’t looked into intra / inter node communication and I’m unsure of its feasibility given my limited recent experience with Flux and Lux. As I’m not a professional developer, it’s likely better for you to handle the integration of other backends. I will update you if I find any meaningful information regarding extended communication on AMD. To better understand the model, today, I drafted a very short … PRIMER, which I’ll review and try to post here as a PDF by the end of the week. While I’m happy to discuss hardware technicalities, I cannot comment on your second paragraph. As indicated previously, my understanding is that the best thing you can do now is set up some form of a legal framework for this project, otherwise, it may end up as an exercise. However, I might be very wrong.
Great, it is merged, I can’t do a dynamic test, just read over it and made some changes to keep the MacOS Metal usage separate from the rest. Glad to see this done too 
thanks!!
Legal… 
like what needs to be done??
Ok, so I will look to get a AMD gpu to work on it as well! great idea
if you are interested let’s make a specific topic for llms in julia
Here is the primer: link. I just finished it and, to be honest, did not have time to check it for correctness. It is marked as a draft.
Regarding your question. I guess something like this. However, I’m sorry, I’m not in a position to advise you on legal topics.
As for AMD. I just did it for a quick testing. It was only for a single GPU. When doing it, my intention was not to make any GitHub commits.
Regarding your proposal, the list looks quite long, and I’m not sure how realistic it is in the short to medium time-frame. I’m also unsure whether Julia is the best choice for some items, or how rational it would be to compete with some of the existing proprietary or open-source solutions. That said, I guess, an LLM should definitely be possible, and your package looks like a great start.
EDIT: 1. Updated the file to VER.0.1.B., 2. Updated to VER.0.1.C., 3. Updated to VER.0.1.D.
Thank you! I think many of us share Richard Feynman’s:
“If you can’t explain something to a first-year student, then you haven’t really understood it.”
and Andrej Karpathy’s:
“If you can’t build it, you don’t understand it.”
There are several excellent “Transformers from scratch” resources—for example, Brandon Rohrer’s guide is very approachable and full of examples.
With @mantzaris’ pure Julia implementation, we can actually step through the code, inspect what every part does, and change anything we want. That makes it much easier to understand how transformers work.
It would be nice to see it documented in the spirit of SICP, with the explanation tied to the code through literate programming. I think that would make it an even better resource for learning, experimentation, and perhaps even research.
I’m glad you like it! I agree, Brandon Rohrer’s guide is probably one of the best and very approachable, so I’ve added it to the references. I also changed the title to “PRIMER: TOWARDS JULIA LARGE LANGUAGE MODELS”. The link to the updated file is in post #25.
And how about the volleyball? :- ) P.S. By the way, I am currently in contact with @Palli, who reached out to me regarding my comments about the MI250X in this thread. According to his research (and you know what he’s capable of in this field), the card is potentially far more capable than I initially anticipated, particularly regarding AITER, TRITON, and support for various quantization methods.
Fantastic to hear! I will try to put in an order for an amd card in the coming months to see it my self too.
Not much sports watching… Occasionally I watch USA football/basket ball when at a sports bar showing it on the big screens
Cool. I like football, volleyball, hockey, squash, and sailing the most. To sum up my part: I don’t know how useful I might be for this potential project. As I’ve reiterated several times on this forum, I’m not a professional coder. Nevertheless, in case I might be of any help, I’m confirming my preliminary expression of interest again. My understanding is that the kind of projects associated with topics like in this thread, even at small to moderate scale, require relatively vast amounts of computing power. I’m currently using a small cluster thanks to the great courtesy of one of the institutions located in your country. The cluster is based on MI250X. I also know that my country (I’m from one of the small / medium-sized countries located in CEE, the EU member) has a similar cluster based on MI250X at its disposal. That’s why I mentioned this card in this thread. So, in case of your potential interest, we might try to contact one of the institutions in my country in order to increase potential computing capabilities a bit. Based on my experience, usually the hardest is to get started. Nevertheless, to be honest, I don’t know how relevant this might be for this potential project. However, this is my best attempt to be helpful. I’d also like to make a disclaimer, that this is not a forward looking statement and really I can’t promise anything. I also hope I didn’t write anything in this thread that is out of place - according to the best of my current knowledge all is fine - however, in case of any remarks, please let me know by a direct message and I will correct immediately. Cheers.
Consider using Muon (PR seems ready, or stuck?):
or (I didn’t know of this, newer/better variant?):
You currently use Adam, why not AdamW (or Muon etc)?
I suppose you’re using SwiGLU (SwiGLU-Clip was new to me and), newer to consider:
I suppose your problem is the 512 token context window, one reason and/or small/bad training dataset(?), for not being better. I think it might be intentional because of quadratic attention. Too costly for you to increase? I suppose SubQ is too hard to implement. You can have some linear layers, if you don’t already (and/or Mamba layers).
This paper seems awesome, but I guess a premature to add to your design:
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7170898
We propose Hierarchical Adaptive Memory (HAM), a memory layer that tiers information into working, episodic, and semantic stores by a utility score (frequency, reuse, recency, novelty, predictive utility, stability) and consolidates frequent, redundant content into fewer prototypes while retaining rare content at higher fidelity. This predictive-coding-inspired compression lowers byte footprint at a similar fine-tuning budget. Unlike prior tokenization-level hierarchical memory, HAM operates at the prototype level and so can serve as an external store, a KV-cache compressor, or an internal memory block.
AVQ-Attention: Adaptive Vector-Quantized Attention
https://arxiv.org/pdf/2607.12789v1
seems like a very important paper (for you/LLMs too? It’s validated on image classification, AI suggested to me it would work for LLMs, and I challenged that, since not mentioned, only LLMs in sources, it thinks it translates to LLMs). If the I understand correctly and the codebook, M, is small, then much better than the quadratic attention I suppose you use, i.e. it’s O(MN).
CommVQ paper is also intriguing, maybe it can combine with above:
.. FP16 KV cache size by 87.5% with 2-bit quantization, while outperforming state-of-the-art KV cache quantization methods. Notably, it enables 1-bit KV cache quantization with minimal accuracy loss, allowing a LLaMA-3.1 8B model to run with a 128K context length on a single RTX 4090 GPU. The source code is available at: GitHub - UMass-Embodied-AGI/CommVQ: [ICML 2025] CommVQ: Commutative Vector Quantization for KV Cache Compression · GitHub.
In case you or anyone wants to do 2-bit quantized models (a bit premature until your generated model is better):
Experiments across diverse LLMs demonstrate that LC-QAT consistently outper-
forms state-of-the-art QAT methods while using only 0.1%–10% of the training data.
They use AdamW in the paper (I suppose your Adam would also work, and probably Muon or similar too).
Note, the paper was rejected (it seems very intriguing to me, found it reference in breakthrough paper below, your call if you want to implement it), but read arguments and answers, still starting this way in the metareview:
I agree with the reviewers that the idea of the paper is simple, elegant, and appealing. Adding a latent variable to the decoder with very small overhead is attractive, and the paper shows improvements on several benchmarks. In this sense, the work is a tangible contribution and shows promising results.
Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation
Autoregressive LLMs have proven the hardest case, for the reasons discussed in our limitations (Section 7)
[..]
The Free Transformer [68] conditions a decoder on a latent variable inferred by a VAE, which is exactly the kind of latent exploration searches over, and training it with Explorative Modeling instead would remove the VAE entirely, along with the exposure bias of training on inferred latents (Section 3.1). Beyond language models, few-step models such as MeanFlow [15] are a natural fit for exploration as well, since exploration can supply the generative expressivity their shortened trajectories amortize. We also believe in combining XMs with Energy-Based Transformers (EBTs) [69], where the biggest documented challenge with EBTs has been end-to-end generation and handling highly multimodal distributions, which is exactly what XMs enable. Paired together, XMs and EBTs could enable more dynamic reasoning, search, and generalization over entire sequences.
[Note e.g. there Figure 9 needs light mode, broke in dark mode]
The longer a model generates, the worse things get—you’ve probably seen this firsthand with video generation models that melt into mush after ten seconds, or LLMs that get less coherent over really long generations.
TLDR: We outscale (feed-forward) transformers while generalizing reasoning/system 2 thinking to any modality/problem without requiring verifiable rewards
. Energy-Based Transformers are the first approach to outscale feed-forward transformers across modalities and with respect to several axes including data, depth, parameters, FLOPs, etc. Energy-Based Transformers can think over every single prediction (i.e. every token in language modeling) and generalize better than existing models.
https://arxiv.org/pdf/2507.02092
Specifically, we train Energy-Based Transformers (EBTs)—a new class of Energy-Based Models (EBMs)—to assign an energy (un-
normalized probability) value to every input and candidate-prediction pair, enabling
predictions through gradient descent-based energy minimization until convergence.
This formulation enables System 2 Thinking to emerge from unsupervised learn-
ing, making it modality and problem agnostic. Across both discrete (text) and
continuous (visual) modalities, we find EBTs scale faster than the dominant Trans-
former++ approach during training, achieving an up to 35% higher scaling rate with
respect to data, batch size, parameters, FLOPs, and depth. During inference, EBTs
improve performance with System 2 Thinking (i.e., extra computation) by 29%
more than the Transformer++ on language tasks, and EBTs outperform Diffusion
Transformers on image denoising while using fewer forward passes
Great material and fantastic suggestions
@Palli, what is your take on [2512.04695] TRINITY: An Evolved LLM Coordinator and [2512.04388] Learning to Orchestrate Agents in Natural Language with the Conductor ? And could [2404.16130] From Local to Global: A Graph RAG Approach to Query-Focused Summarization and [2506.05690] When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation be related?
Also, does LINK still hold up? I mean broadly, not “OOMs” as if I’m recalling correctly, Mr. Dario recently told Mr. Dwarkesh that “we are near the end of exponential”.
What will the future look like? Will we end up with a handful of giant models/libraries, or will we move toward smaller, more specialized models, akin to library departments? Or perhaps both?
EDIT: Hey @Palli, I noticed another private message, this time related to my recent public post. I’d like to kindly ask you to reply publicly instead. I’ll try to reply to all your private messages this evening. I’m always happy to make friends and chat privately. However, I’d prefer to avoid situations where public discussions are moved into a private domain, as it puts me in a position where I don’t feel comfortable. I’m aware that there are situations where private communication is required and can supersede a public discussion, but I don’t think that is the case here. I really don’t see a need for that in this particular case. Thank you for the understanding.
I realize now you most likely do not have 3 weight matrices per layer (and SwiGLU), rather 2 (and GELU). But that is the old still current design, that Kimi K3 still uses (though it switched out the mainstream SwiGLU). I would rather do, since you asked me;
TL;DR: We introduce STEM: a static, token-indexed sparse architecture design that swaps FFN up-projection for a layer-local embedding lookup. Despite extreme sparsity, STEM trains stably, stores more parametric knowlege while speeding up FFN layers by 3x. More interestingly, STEM improves interpretability compared to the dense and exisiting sparse baselines and strengthens long-context scaling; across 350M–1B, yields up to ~3–4% accuracy gains across various knowledge based and reasoning downstream tasks including ARC-Challenge, OpenbookQA, GSM8K, MMLU, Big-Bench Hard, etc.
Based on the results of our ablation studies, we introduce STEM, a static, token-indexed sparse architecture design that swaps FFN up-projection for a layer-local embedding lookup. It is critical to leave the gating path in FFN unchanged to preserve the contextual ability of the model.
I found that paper referenced here under 5.1 Per-Layer Embedding, but even more intriguing:
5.2 Per-Layer N-gram Embedding
Building upon PLE, we propose Per-Layer N-gram Embedding (PLNE), a novel extension that replaces the base embedding outputs with N-gram Embedding outputs at each layer, thereby enabling more flexible and targeted parameter scaling within the MoE framework.
I doubt you have MoE yet, but that can be deferred, and I’m not sure if this and PLNE depends on it. They work together, but I think the order you implement or even if you skip MoE it might work, if you want to trust me.
Kimi K3 switched out SwiGLU for better training stability, I believe it’s a synonym for SiLU, what you see in the papers/formulas, and PowLU is another option, now I’m no longer sure it’s better than what Kimi K3 chose, or which is newer. It might not matter, you can chose either.
AI answer:
Yes, Kimi K3 features traditional gate, up, and down projection matrices within its dense MLPs and expert blocks, but wraps them in a compressed “Stable LatentMoE” routing mechanism.
..
Activation Twist: Integrates a modified activation variant called SiTU-GLU (Sigmoid Tanh Unit GLU) rather than a pure SwiGLU to soft-cap activation magnitudes across the deep chain of projections
I had suggested FlashAttention, but that means you need KV cache first, and you’re not yet there. But the paper(s) above may change that picture.
I’m wondering, what is your opinion on Mamba2 and Hybrid Models? I recall that in one of your posts in the initial thread on this topic, you were rather skeptical about them. I can’t stop thinking about it. The Mamba ad on YT is just state of the art. And the fact that NVIDIA is into it is a bit … intimidating. What’s your take on it, if I may ask?
I just learned yesterday about a new paper, “BDH-CQ: In-Context Learning with Recurrent Latent Reasoning”, and consequently about its predecessor, “The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain” . These papers propose a new architecture family called BDH-CQ. I have added both to the list of references in the file I linked in post #25.
I find them interesting for various reasons. One that caught my particular attention is that the original paper proposes a scale-free, biologically inspired, attention-based state-space sequence-learning architecture. Furthermore, the latest paper refers more than once to the argument discussed in Quanta Magazine’s interview with Dr. Evelina Fedorenko that language processing and general reasoning in the human brain may rely on at least partially distinct neural systems (I included this Quanta Magazine article in my initial list of references).
So, what is this new jazz all about? As I understand it currently, the basic idea of the latest paper is that the model can “think” internally many times before speaking once. In other words, the model solves problems through iterative computation in a high-dimensional latent space without verbalizing intermediate reasoning.
From IBM Research in May:
https://arxiv.org/pdf/2601.21766
We design novel architectural components based on this function class that can replace Multi-
head Attention and Feed-Forward Networks in Transformer blocks while requiring
much fewer parameters. .. Our components are a plug-in replacement requiring
little change in training or inference procedures that have already been put in
place for Transformer-based models thus making our approach easy to incorporate
in large industrial workflows. We experiment on two very different transformer
architectures GPT2-xl (1.5B) and Llama3 (3.2B), where the former we pre-train on
OpenWebText and GneissWeb, while the latter we pre-train on the docling data
mix which consists of nine different datasets. Results show that the performance
on downstream classification, Q& A, reasoning and text understanding tasks of
our models is competitive and sometimes even superior to the original models
with 2/3 to 1/2 the parameters and shorter pre-training time. We believe that future
implementations customized to hardware will further bring out the true potential of
our architectures.
where, aks are complex numbers, CoFrNets [32] were introduced for supervised learning problems
..
This simple architecture was shown to have universal approximation capabilities when we ensemble enough of these ladders. However, .. We address the above challenges in this paper by making the following contributions that distinguish it significantly from [32]:
- We propose novel continued fraction architectures for (causal) attention and FFNs as depicted in Figure 1. We call our architecture with both components replaced as Continued Fraction Generative Network (CoFrGeNet). We report results replacing either FFN or attention or both offering the possibility to the user of replacing only one or both of the components for their application. Even replacing one component can offer significant parameter and training time savings as seen in our experiments.
It references:
Interpretable neural architecture inspired by continued fractions. In M. Ranzato, A. Beygelzimer,
Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing
Systems, volume 34, pages 21668–21680. Curran Associates, Inc., 2021
It references Evaluating Long Context (Reasoning) Ability | wh "What do 1M and 500K context windows have in common? They are both actually 64K. What do 1M and 500K context windows have in common? They are both actually 64K. and I don’t know that do be outdated, so I would look more into RLMs and reimplement (or reuse?). Its paper was updated in May:
GitHub - alexzhang13/rlm: General plug-and-play inference library for Recursive Language Models (RLMs), supporting various sandboxes. · GitHub
And FYI: I already posted my related to harnesses in another thread:
Our best results use Opus 5 in Prime Agent to achieve 95.5% RHAE Best@1, which surpasses the ARC reported human expert baseline of 95.4%.
[I’ve since seen others claim up to 99%]
Do you want me to add any papers to the list of references of the primer?
I’d definitely be interested in seeing a few of these papers added to the primer, especially the work on continued-fraction architectures and Recursive Language Models. They seem quite relevant to the direction of the project, and having them in the references would make it easier to explore the ideas further.