# Proof of concept LLM chatbot built with Julia: KeemenaLM.jl

**URL:** <https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961>\
**Category:** Package Announcements\
**Created:** [July 5, 2026, 8:37pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961 "2026-07-05T20:37:34Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![anon28281882](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon28281882](https://discourse.julialang.org/u/anon28281882)\
**Post date:** [July 23, 2026, 11:28pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/21 "2026-07-23T23:28:33Z")

</div>

Similar to @pitsianis, I’ve added and tested preliminary `AMDGPU` support. I haven’t looked into intra / inter node communication and I’m unsure of its feasibility given my limited recent experience with `Flux` and `Lux`. As I’m not a professional developer, it’s likely better for you to handle the integration of other backends. I will update you if I find any meaningful information regarding extended communication on AMD. To better understand the model, today, I drafted a very short … `PRIMER`, which I’ll review and try to post here as a PDF by the end of the week. While I’m happy to discuss hardware technicalities, I cannot comment on your second paragraph. As indicated previously, my understanding is that the best thing you can do now is set up some form of a legal framework for this project, otherwise, it may end up as an exercise. However, I might be very wrong.

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [July 24, 2026, 9:16pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/22 "2026-07-24T21:16:56Z")

</div>

Great, it is merged, I can’t do a dynamic test, just read over it and made some changes to keep the MacOS Metal usage separate from the rest. Glad to see this done too 😃 thanks!!

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [July 24, 2026, 9:18pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/23 "2026-07-24T21:18:45Z")

</div>

> [@anon28281882](#):
>
> I’ve added and tested preliminary `AMDGPU` support. I haven’t looked into intra / inter node communication and I’m unsure of its feasibility given my limited recent experience with `Flux` and `Lux`. As I’m not a professional developer, it’s likely better for you to handle the integration of other backends. I will update you if I find any meaningful information regarding extended communication on AMD. To better understand the model, today, I drafted a very short … `PRIMER`, which I’ll review and try to post here as a PDF by the end of the week. While I’m happy to discuss hardware technicalities, I cannot comment on your second paragraph. As indicated previously, my understanding is that the best thing you can do now is set up some form of a legal framework for this project, otherwise, it may end up as an exercise. However, I might be very wrong.

Legal… 😅 like what needs to be done??

Ok, so I will look to get a AMD gpu to work on it as well! great idea

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [July 25, 2026, 6:13pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/24 "2026-07-25T18:13:04Z")

</div>

> [@Proposal: LLMs and Generative AI](https://discourse.julialang.org/t/proposal-llms-and-generative-ai/138463):
>
> Julia for LLMs and Generative AI I wanted to see what the community thinks about creating a dedicated LLMs and Generative AI section under the Domains category. Working on: Training foundation models Fine tuning and alignment Inference and optimization Local LLM deployment GPU acceleration Agent systems RAG Embeddings and vector databases Prompt engineering Evaluation and benchmarking Multimodal models Julia wrappers around existing ecosystems New Julia-native LLM frameworks and tools Testing…

if you are interested let’s make a specific topic for llms in julia

---

<div class="post-metadata">

**Author:** ![anon28281882](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon28281882](https://discourse.julialang.org/u/anon28281882)\
**Post date:** [July 27, 2026, 12:47am UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/25 "2026-07-27T00:47:06Z")

</div>

Here is the primer: [link](https://github.com/justcodebase/documents/blob/main/PRIMER--TOWARDS-JULIA-LARGE-LANGUAGE-MODELS.pdf). I just finished it and, to be honest, did not have time to check it for correctness. It is marked as a draft.

Regarding your question. I guess something like this. However, I’m sorry, I’m not in a position to advise you on legal topics.

As for AMD. I just did it for a quick testing. It was only for a single GPU. When doing it, my intention was not to make any GitHub commits.

Regarding your proposal, the list looks quite long, and I’m not sure how realistic it is in the short to medium time-frame. I’m also unsure whether Julia is the best choice for some items, or how rational it would be to compete with some of the existing proprietary or open-source solutions. That said, I guess, an LLM should definitely be possible, and your package looks like a great start.

EDIT: 1. Updated the file to VER.0.1.B., 2. Updated to VER.0.1.C., 3. Updated to VER.0.1.D.

---

<div class="post-metadata">

**Author:** ![pitsianis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pitsianis/32/26588_2.png) [@pitsianis](https://discourse.julialang.org/u/pitsianis)\
**Post date:** [July 28, 2026, 2:31am UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/26 "2026-07-28T02:31:36Z")

</div>

> [@anon28281882](#):
>
> Here is the primer:

Thank you! I think many of us share Richard Feynman’s:

> _“If you can’t explain something to a first-year student, then you haven’t really understood it.”_

and Andrej Karpathy’s:

> _“If you can’t build it, you don’t understand it.”_

There are several excellent “Transformers from scratch” resources—for example, [Brandon Rohrer’s guide](https://brandonrohrer.com/transformers.html) is very approachable and full of examples.

With @mantzaris’ pure Julia implementation, we can actually step through the code, inspect what every part does, and change anything we want. That makes it much easier to understand how transformers work.

It would be nice to see it documented in the spirit of [SICP](https://mitp-content-server.mit.edu/books/content/sectbyfn/books_pres_0/6515/sicp.zip/index.html), with the explanation tied to the code through literate programming. I think that would make it an even better resource for learning, experimentation, and perhaps even research.

---

<div class="post-metadata">

**Author:** ![anon28281882](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon28281882](https://discourse.julialang.org/u/anon28281882)\
**Post date:** [July 28, 2026, 4:06pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/27 "2026-07-28T16:06:54Z")

</div>

I’m glad you like it! I agree, Brandon Rohrer’s guide is probably one of the best and very approachable, so I’ve added it to the references. I also changed the title to “PRIMER: TOWARDS JULIA LARGE LANGUAGE MODELS”. The link to the updated file is in post #25.

---

<div class="post-metadata">

**Author:** ![anon28281882](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon28281882](https://discourse.julialang.org/u/anon28281882)\
**Post date:** [August 2, 2026, 2:29pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/28 "2026-08-02T14:29:32Z")

</div>

> [@mantzaris](#):
>
> I don’t watch football/soccer, so whoever puts on the best performance 🙂

And how about the volleyball? :- ) P.S. By the way, I am currently in contact with @Palli, who reached out to me regarding my comments about the MI250X in this thread. According to his research (and you know what he’s capable of in this field), the card is potentially far more capable than I initially anticipated, particularly regarding AITER, TRITON, and support for various quantization methods.

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [August 2, 2026, 8:27pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/29 "2026-08-02T20:27:28Z")

</div>

> [@anon28281882](#):
>
> ut to me regarding my comments about the MI250X in this thread. According to his research (and you know what he’s capable of in this field), the card is potentially far more capable than I initially anticipated, particularly regarding AITER, TRITON, and support for various quantization methods.

Fantastic to hear! I will try to put in an order for an amd card in the coming months to see it my self too.

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [August 2, 2026, 8:28pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/30 "2026-08-02T20:28:33Z")

</div>

Not much sports watching… Occasionally I watch USA football/basket ball when at a sports bar showing it on the big screens

---

<div class="post-metadata">

**Author:** ![anon28281882](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon28281882](https://discourse.julialang.org/u/anon28281882)\
**Post date:** [August 2, 2026, 10:25pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/31 "2026-08-02T22:25:56Z")

</div>

Cool. I like football, volleyball, hockey, squash, and sailing the most. To sum up my part: I don’t know how useful I might be for this potential project. As I’ve reiterated several times on this forum, I’m not a professional coder. Nevertheless, in case I might be of any help, I’m confirming my preliminary expression of interest again. My understanding is that the kind of projects associated with topics like in this thread, even at small to moderate scale, require relatively vast amounts of computing power. I’m currently using a small cluster thanks to the great courtesy of one of the institutions located in your country. The cluster is based on MI250X. I also know that my country (I’m from one of the small / medium-sized countries located in CEE, the EU member) has a similar cluster based on MI250X at its disposal. That’s why I mentioned this card in this thread. So, in case of your potential interest, we might try to contact one of the institutions in my country in order to increase potential computing capabilities a bit. Based on my experience, usually the hardest is to get started. Nevertheless, to be honest, I don’t know how relevant this might be for this potential project. However, this is my best attempt to be helpful. I’d also like to make a disclaimer, that this is not a forward looking statement and really I can’t promise anything. I also hope I didn’t write anything in this thread that is out of place - according to the best of my current knowledge all is fine - however, in case of any remarks, please let me know by a direct message and I will correct immediately. Cheers.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [August 2, 2026, 11:58pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/32 "2026-08-02T23:58:42Z")

</div>

Consider using Muon (PR seems ready, or stuck?):

> <https://github.com/FluxML/Optimisers.jl/pull/203>
>
> This adds Muon (https://kellerjordan.github.io/posts/muon/), which uses an appro…ximate orthogonalization before the update. There isn't a publication, but it gave a key improvement in the nanoGPT training "speedrun" attempt: https://github.com/KellerJordan/modded-nanogpt
> 
> This was started by @mashu 
> 
> \### PR Checklist
> 
> \- \[\] Tests are added
> \- \[\] Documentation, if applicable

> **[To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters](https://arxiv.org/abs/2603.00742v1)**
>
> While Adam has long been the ubiquitous default optimizer for deep neural networks, Muon has recently seen rapid adoption due to its superior training speed. Although much of the literature focuses on validating the benefits of Muon, our work...

or (I didn’t know of this, newer/better variant?):

> [@Has anyone tried implementing the LiMuon Optimizer?](https://discourse.julialang.org/t/has-anyone-tried-implementing-the-limuon-optimizer/132742):
>
> Has anyone tried implementing the [LiMuon optimizer](https://arxiv.org/abs/2509.14562)? I made an issue [here](https://github.com/FluxML/Optimisers.jl/issues/213) in the Optimisers.jl package but thought I’d ask here as well if someone has implemented it elsewhere? Also any hands on experience with the method itself would be interesting to hear about. According to the paper it beats AdamW in a few benchmarks both on training/testing error and convergence speed.

You currently use Adam, why not AdamW (or Muon etc)?

I suppose you’re using SwiGLU (SwiGLU-Clip was new to me and), newer to consider:

> **[PowLU: An Activation Function for Stable Pre-Training of LLMs](https://arxiv.org/abs/2605.25704)**
>
> In contemporary large language models (LLMs), the swish-gated linear unit (SwiGLU) activation function is widely adopted to regulate the information flow and introduce non-linearity. For large positive inputs, SwiGLU approximates the quadratic...

I suppose your problem is the 512 token context window, one reason and/or small/bad training dataset(?), for not being better. I think it might be intentional because of quadratic attention. Too costly for you to increase? I suppose SubQ is too hard to implement. You can have some linear layers, if you don’t already (and/or Mamba layers).

This paper seems awesome, but I guess a premature to add to your design:

[https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=7170898](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7170898)

> We propose Hierarchical Adaptive Memory (HAM), a memory layer that tiers information into working, episodic, and semantic stores by a utility score (frequency, reuse, recency, novelty, predictive utility, stability) and consolidates frequent, redundant content into fewer prototypes while retaining rare content at higher fidelity. This predictive-coding-inspired compression lowers byte footprint at a similar fine-tuning budget. Unlike prior tokenization-level hierarchical memory, HAM operates at the prototype level and so can serve as an external store, a KV-cache compressor, or an internal memory block.

https://embed.reddit.com/r/LocalLLaMA/comments/1v3nw3j/cactus_hybrid_we_taught_gemma_4_to_know_when_its/?embed=true&ref_source=embed&ref=share

AVQ-Attention: Adaptive Vector-Quantized Attention  
[https://arxiv.org/pdf/2607.12789v1](https://arxiv.org/pdf/2607.12789v1)

seems like a very important paper (for you/LLMs too? It’s validated on image classification, AI suggested to me it would work for LLMs, and I challenged that, since not mentioned, only LLMs in sources, it thinks it translates to LLMs). If the I understand correctly and the codebook, M, is small, then much better than the quadratic attention I suppose you use, i.e. it’s O(MN).

CommVQ paper is also intriguing, maybe it can combine with above:

> .. FP16 KV cache size by 87.5% with 2-bit quantization, while outperforming state-of-the-art KV cache quantization methods. Notably, it enables 1-bit KV cache quantization with minimal accuracy loss, allowing a LLaMA-3.1 8B model to run with a 128K context length on a single RTX 4090 GPU. The source code is available at: [GitHub - UMass-Embodied-AGI/CommVQ: [ICML 2025] CommVQ: Commutative Vector Quantization for KV Cache Compression · GitHub](https://github.com/UMass-Embodied-AGI/CommVQ).

In case you or anyone wants to do 2-bit quantized models (a bit premature until your generated model is better):

> **[GitHub - AI9Stars/UniSVQ: \[ICML 2026\] Codes of paper UniSVQ: 2-bit Unified...](https://github.com/AI9Stars/UniSVQ)**
>
> \[ICML 2026\] Codes of paper UniSVQ: 2-bit Unified Scalar-Vector Quantization and LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization.

> Experiments across diverse LLMs demonstrate that LC-QAT consistently outper-  
> forms state-of-the-art QAT methods while using only 0.1%–10% of the training data.

They use AdamW in the paper (I suppose your Adam would also work, and probably Muon or similar too).

https://embed.reddit.com/r/singularity/comments/1odklk9/meta_the_free_transformer_an_improvement_to/?embed=true&ref_source=embed&ref=share

Note, the paper was rejected (it seems very intriguing to me, found it reference in breakthrough paper below, your call if you want to implement it), but read arguments and answers, still starting this way in the metareview:

> **[Verifying your browser | OpenReview](https://openreview.net/challenge?redirect=%2Fforum%3Fid%3DRV43b2B1SC)**

> I agree with the reviewers that the idea of the paper is simple, elegant, and appealing. Adding a latent variable to the decoder with very small overhead is attractive, and the paper shows improvements on several benchmarks. In this sense, the work is a tangible contribution and shows promising results.

> **[paper.pdf](https://explorative-modeling.github.io/static/pdfs/paper.pdf)**
>
> 4.49 MB

Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

> Autoregressive LLMs have proven the hardest case, for the reasons discussed in our limitations (Section 7)  
> [..]  
> The Free Transformer [68] conditions a decoder on a latent variable inferred by a VAE, which is exactly the kind of latent exploration searches over, and training it with Explorative Modeling instead would remove the VAE entirely, along with the exposure bias of training on inferred latents (Section 3.1). Beyond language models, few-step models such as MeanFlow [15] are a natural fit for exploration as well, since exploration can supply the generative expressivity their shortened trajectories amortize. We also believe in combining XMs with Energy-Based Transformers (EBTs) [69], where the biggest documented challenge with EBTs has been end-to-end generation and handling highly multimodal distributions, which is exactly what XMs enable. Paired together, XMs and EBTs could enable more dynamic reasoning, search, and generalization over entire sequences.

> **[Alexi Gladstone | Explorative Modeling -- Unlocking a Third Pretraining Axis...](https://alexiglad.github.io/blog/2026/explorative_modeling/)**
>
> What in the world is Explorative Modeling?

[Note e.g. there Figure 9 needs light mode, broke in dark mode]

> **[GitHub - alexiglad/XM: PyTorch Code for Explorative Modeling: Unlocking a...](https://github.com/alexiglad/XM)**
>
> PyTorch Code for Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation

> The longer a model generates, the worse things get—you’ve probably seen this firsthand with video generation models that melt into mush after ten seconds, or LLMs that get less coherent over really long generations.

> **[Alexi Gladstone | Energy-Based Transformers are Scalable Learners and Thinkers](https://alexiglad.github.io/blog/2025/ebt/)**
>
> What are Energy-Based Transformers and why should I care about them?

> **TLDR** : We **outscale** (feed-forward) transformers while **generalizing** reasoning/system 2 thinking to any modality/problem **without** requiring verifiable rewards​😮. Energy-Based Transformers are the **first approach** to outscale feed-forward transformers across modalities and with respect to several axes including data, depth, parameters, FLOPs, etc. Energy-Based Transformers can think over every single prediction (i.e. every token in language modeling) and generalize better than existing models.

[https://arxiv.org/pdf/2507.02092](https://arxiv.org/pdf/2507.02092)

> Specifically, we train **Energy-Based Transformers (EBTs)**—a new class of Energy-Based Models (EBMs)—to assign an energy (un-  
> normalized probability) value to every input and candidate-prediction pair, enabling  
> predictions through gradient descent-based energy minimization until convergence.  
> This formulation enables System 2 Thinking to emerge from unsupervised learn-  
> ing, making it modality and problem agnostic. Across both discrete (text) and  
> continuous (visual) modalities, we find EBTs scale faster than the dominant Trans-  
> former++ approach during training, achieving an up to 35% higher scaling rate with  
> respect to data, batch size, parameters, FLOPs, and depth. During inference, EBTs  
> improve performance with System 2 Thinking (i.e., extra computation) by 29%  
> more than the Transformer++ on language tasks, and EBTs outperform Diffusion  
> Transformers on image denoising while using fewer forward passes

> **[Mean Flows for One-step Generative Modeling](https://arxiv.org/abs/2505.13447)**
>
> We propose a principled and effective framework for one-step generative modeling. We introduce the notion of average velocity to characterize flow fields, in contrast to instantaneous velocity modeled by Flow Matching methods. A well-defined identity...

> **[MODUS: Decoder-only Any-to-Any Modeling of Diverse Modalities](https://modus-multimodal.epfl.ch/)**
>
> MODUS unifies any-to-any multimodal generation in a single decoder-only model.

---

<div class="post-metadata">

**Author:** ![mantzaris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mantzaris/32/3852_2.png) [@mantzaris](https://discourse.julialang.org/u/mantzaris)\
**Post date:** [August 3, 2026, 12:55am UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/33 "2026-08-03T00:55:47Z")

</div>

Great material and fantastic suggestions

---

<div class="post-metadata">

**Author:** ![anon28281882](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon28281882](https://discourse.julialang.org/u/anon28281882)\
**Post date:** [August 3, 2026, 2:22am UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/34 "2026-08-03T02:22:56Z")

</div>

@Palli, what is your take on [[2512.04695] TRINITY: An Evolved LLM Coordinator](https://arxiv.org/abs/2512.04695) and [[2512.04388] Learning to Orchestrate Agents in Natural Language with the Conductor](https://arxiv.org/abs/2512.04388) ? And could [[2404.16130] From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://arxiv.org/abs/2404.16130) and [[2506.05690] When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation](https://arxiv.org/abs/2506.05690) be related?

Also, does [LINK](https://situational-awareness.ai/wp-content/uploads/2024/06/situationalawareness.pdf) still hold up? I mean broadly, not “OOMs” as if I’m recalling correctly, Mr. Dario recently told Mr. Dwarkesh that “[we are near the end of exponential](https://www.dwarkesh.com/p/dario-amodei-2)”.

What will the future look like? Will we end up with a handful of giant models/libraries, or will we move toward smaller, more specialized models, akin to library departments? Or perhaps both?

EDIT: Hey @Palli, I noticed another private message, this time related to my recent public post. I’d like to kindly ask you to reply publicly instead. I’ll try to reply to all your private messages this evening. I’m always happy to make friends and chat privately. However, I’d prefer to avoid situations where public discussions are moved into a private domain, as it puts me in a position where I don’t feel comfortable. I’m aware that there are situations where private communication is required and can supersede a public discussion, but I don’t think that is the case here. I really don’t see a need for that in this particular case. Thank you for the understanding.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [August 9, 2026, 7:46pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/35 "2026-08-09T19:46:05Z")

</div>

I realize now you most likely do not have 3 weight matrices per layer (and SwiGLU), rather 2 (and GELU). But that is the old still current design, that Kimi K3 still uses (though it switched out the mainstream SwiGLU). I would rather do, since you asked me;

> **[STEM](https://infini-ai-lab.github.io/STEM/)**
>
> STEM: Scaling Transformers with Embedding Modules

> **TL;DR:** We introduce **STEM** : a **static, token-indexed** sparse architecture design that swaps FFN up-projection for a **layer-local embedding lookup**. Despite extreme sparsity, STEM trains **stably** , stores **more parametric knowlege** while **speeding up FFN layers by 3x**. More interestingly, STEM improves **interpretability** compared to the dense and exisiting sparse baselines and strengthens **long-context scaling** ; across 350M–1B, yields up to **~3–4%** accuracy gains across various knowledge based and reasoning downstream tasks including ARC-Challenge, OpenbookQA, GSM8K, MMLU, Big-Bench Hard, etc.

> Based on the results of our ablation studies, we introduce **STEM** , a _static, token-indexed_ sparse architecture design that **swaps FFN up-projection** for a layer-local embedding lookup. _It is critical to leave the gating path in FFN unchanged to preserve the contextual ability of the model._

I found that paper referenced here under 5.1 Per-Layer Embedding, but even more intriguing:

> **[Scaling Embeddings Outperforms Scaling Experts in Language Models](https://arxiv.org/abs/2601.21204)**
>
> While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embedding scaling as a potent,...

> **5.2 Per-Layer N-gram Embedding**  
> Building upon PLE, we propose Per-Layer N-gram Embedding (PLNE), a novel extension that replaces the base embedding outputs with N-gram Embedding outputs at each layer, thereby enabling more flexible and targeted parameter scaling within the MoE framework.

I doubt you have MoE yet, but that can be deferred, and I’m not sure if this and PLNE depends on it. They work together, but I think the order you implement or even if you skip MoE it might work, if you want to trust me.

Kimi K3 switched out SwiGLU for better training stability, I believe it’s a synonym for SiLU, what you see in the papers/formulas, and PowLU is another option, now I’m no longer sure it’s better than what Kimi K3 chose, or which is newer. It might not matter, you can chose either.

> **[SwiGLU in modern LLMs](https://sebastianraschka.com/faq/docs/swiglu-modern-llms.html)**
>
> SwiGLU uses a SiLU-activated gate to modulate a second feed-forward projection, adding a multiplicative interaction while controlling cost through the interm...

> **[PowLU: An Activation Function for Stable Pre-Training of LLMs](https://arxiv.org/html/2605.25704v1)**

AI answer:

> Yes, Kimi K3 features traditional gate, up, and down projection matrices within its dense MLPs and expert blocks, but wraps them in a compressed “Stable LatentMoE” routing mechanism.  
> ..  
> Activation Twist: Integrates a modified activation variant called SiTU-GLU (Sigmoid Tanh Unit GLU) rather than a pure SwiGLU to soft-cap activation magnitudes across the deep chain of projections

I had suggested FlashAttention, but that means you need KV cache first, and you’re not yet there. But the paper(s) above may change that picture.

---

<div class="post-metadata">

**Author:** ![anon28281882](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon28281882](https://discourse.julialang.org/u/anon28281882)\
**Post date:** [August 10, 2026, 2:34am UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/36 "2026-08-10T02:34:46Z")

</div>

I’m wondering, what is your opinion on Mamba2 and Hybrid Models? I recall that in one of your posts in the initial thread on this topic, you were rather skeptical about them. I can’t stop thinking about it. The Mamba ad on YT is just state of the art. And the fact that NVIDIA is into it is a bit … intimidating. What’s your take on it, if I may ask?

---

<div class="post-metadata">

**Author:** ![anon28281882](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon28281882](https://discourse.julialang.org/u/anon28281882)\
**Post date:** [August 12, 2026, 6:08pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/37 "2026-08-12T18:08:33Z")

</div>

I just learned yesterday about a new paper, “BDH-CQ: In-Context Learning with Recurrent Latent Reasoning”, and consequently about its predecessor, “The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain” . These papers propose a new architecture family called `BDH-CQ`. I have added both to the list of references in the file I linked in post #25.

I find them interesting for various reasons. One that caught my particular attention is that the original paper proposes a scale-free, biologically inspired, attention-based state-space sequence-learning architecture. Furthermore, the latest paper refers more than once to the argument discussed in Quanta Magazine’s interview with Dr. Evelina Fedorenko that language processing and general reasoning in the human brain may rely on at least partially distinct neural systems (I included this Quanta Magazine article in my initial list of references).

So, what is this new jazz all about? As I understand it currently, the basic idea of the latest paper is that the model can “think” internally many times before speaking once. In other words, the model solves problems through iterative computation in a high-dimensional latent space without verbalizing intermediate reasoning.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [August 17, 2026, 1:39am UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/38 "2026-08-17T01:39:58Z")

</div>

From IBM Research in May:  
[https://arxiv.org/pdf/2601.21766](https://arxiv.org/pdf/2601.21766)

> We design novel architectural components based on this function class that can replace Multi-  
> head Attention and Feed-Forward Networks in Transformer blocks while requiring  
> much fewer parameters. .. Our components are a plug-in replacement requiring  
> little change in training or inference procedures that have already been put in  
> place for Transformer-based models thus making our approach easy to incorporate  
> in large industrial workflows. We experiment on two very different transformer  
> architectures GPT2-xl (1.5B) and Llama3 (3.2B), where the former we pre-train on  
> OpenWebText and GneissWeb, while the latter we pre-train on the docling data  
> mix which consists of nine different datasets. Results show that the performance  
> on downstream classification, Q& A, reasoning and text understanding tasks of  
> our models is competitive and sometimes even superior to the original models  
> with 2/3 to 1/2 the parameters and shorter pre-training time. We believe that future  
> implementations customized to hardware will further bring out the true potential of  
> our architectures.

> where, aks are complex numbers, CoFrNets [32] were introduced for supervised learning problems  
> ..  
> This simple architecture was shown to have universal approximation capabilities when we ensemble enough of these ladders. However, .. We address the above challenges in this paper by making the following contributions that distinguish it significantly from [32]:
> 
> 1. We propose novel continued fraction architectures for (causal) attention and FFNs as depicted in Figure 1. We call our architecture with both components replaced as Continued Fraction Generative Network (CoFrGeNet). We report results replacing either FFN or attention or both offering the possibility to the user of replacing only one or both of the components for their application. Even replacing one component can offer significant parameter and training time savings as seen in our experiments.

It references:  
Interpretable neural architecture inspired by continued fractions. In M. Ranzato, A. Beygelzimer,  
Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing  
Systems, volume 34, pages 21668–21680. Curran Associates, Inc., 2021

> **[Recursive Language Models: the paradigm of 2026](https://www.primeintellect.ai/blog/rlm)**
>
> How we plan to manage extremely long contexts

It references [Evaluating Long Context (Reasoning) Ability | wh](https://nrehiew.github.io/blog/long_context/) "_What do 1M and 500K context windows have in common? They are both actually 64K._ _What do 1M and 500K context windows have in common? They are both actually 64K._ and I don’t know that do be outdated, so I would look more into RLMs and reimplement (or reuse?). Its paper was updated in May:  
[GitHub - alexzhang13/rlm: General plug-and-play inference library for Recursive Language Models (RLMs), supporting various sandboxes. · GitHub](https://github.com/alexzhang13/rlm)

And FYI: I already posted my related to harnesses in another thread:

> **[Prime Agent: A self-improving RLM agent](https://www.primeintellect.ai/blog/prime-agent)**
>
> Prime Agent is our open-source, self-improving coding harness built around two abstractions: the Recursive Language Model (RLM) and the Continual Harness. With Opus 5, it achieves 95.5% on ARC-AGI-3, surpassing the reported human expert baseline.

> Our best results use Opus 5 in Prime Agent to achieve **95.5% RHAE Best@1** , **which surpasses the ARC reported human expert baseline of 95.4%**.

[I’ve since seen others claim up to 99%]

---

<div class="post-metadata">

**Author:** ![anon28281882](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@anon28281882](https://discourse.julialang.org/u/anon28281882)\
**Post date:** [August 17, 2026, 2:41pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/39 "2026-08-17T14:41:58Z")

</div>

Do you want me to add any papers to the list of references of the primer?

---

<div class="post-metadata">

**Author:** ![Myar\_joyia](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/myar_joyia/32/223456_2.png) [@Myar\_joyia](https://discourse.julialang.org/u/Myar_joyia)\
**Post date:** [August 20, 2026, 7:09pm UTC](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961/40 "2026-08-20T19:09:12Z")

</div>

I’d definitely be interested in seeing a few of these papers added to the primer, especially the work on continued-fraction architectures and Recursive Language Models. They seem quite relevant to the direction of the project, and having them in the references would make it easier to explore the ideas further.

[Previous page](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961.md?page=1)

[Next page](https://discourse.julialang.org/t/proof-of-concept-llm-chatbot-built-with-julia-keemenalm-jl/137961.md?page=3)