# Community Interest Check: LLMs from Scratch in Pure Julia

**URL:** <https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796>\
**Category:** Offtopic\
**Tags:** package\
**Created:** [October 26, 2024, 7:10pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796 "2024-10-26T19:10:38Z")\
**Posts on this page:** 1\
**Showing post:** 46

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [August 17, 2025, 8:18pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/46 "2025-08-17T20:18:36Z")

</div>

This seems to be a big deal (to recreated in Julia), and the code open source/Apache 2:

> **[GitHub - sapientinc/HRM: Hierarchical Reasoning Model Official Release](https://github.com/sapientinc/HRM)**
>
> Hierarchical Reasoning Model Official Release

> **[sapientinc (Sapient AI)](https://huggingface.co/sapientinc)**
>
> Org profile for Sapient AI on Hugging Face, the AI community building the future.

> **[The Hidden Drivers of HRM's Performance on ARC-AGI](https://arcprize.org/blog/hrm-analysis)**
>
> We scored on hidden tasks, ran ablations, and found that performance from the Hierarchical Reasoning Model comes from an unexpected source

FYI @mantzaris

The more I read of their excellent paper, then more I want to read and quote, I think it should be obvious to practitioners, this is a game-changer:

**Hierarchical Reasoning Model**  
[https://arxiv.org/pdf/2506.21734](https://arxiv.org/pdf/2506.21734)

> … These results underscore HRM’s potential as a transformative advancement toward universal computation and general-purpose reasoning systems.

> The fixed depth of standard Transformers places them in computational complexity classes such as AC0 or TC0, preventing them from solving problems that require polynomial time. LLMs are not Turing-complete and thus they cannot, at least in a purely end-to-end manner, execute complex algorithmic reasoning that is necessary for deliberate planning or symbolic manipulation tasks
> 
> The LLMs literature has relied largely on Chain-of-Thought (CoT) prompting for reasoning.  
> CoT externalizes reasoning into token-level language by breaking down complex tasks into simpler intermediate steps, sequentially generating text using a shallow model. However, CoT for reasoning is a crutch, not a satisfactory solution. It relies on brittle, human-defined decompositions …  
> A more efficient approach is needed to minimize these data requirements.
> 
> Towards this goal, we explore “latent reasoning”, where the model conducts computations within its internal hidden state space. This aligns with the understanding that language is a tool for human communication, not the substrate of thought itself; the brain sustains lengthy, coherent chains of reasoning with remarkable efficiency in a latent space, without constant translation back to language. However, the power of latent reasoning is still fundamentally constrained by a model’s _effective computational depth_. Naively stacking layers is notoriously difficult due to vanishing gradients, which plague training stability and effectiveness. Recurrent architectures, a natural alternative for sequential tasks, often suffer from early convergence, rendering subsequent computational steps inert, and rely on the biologically implausible, computationally expensive and memory intensive Backpropagation Through Time (BPTT) for training.
> 
> The human brain provides a compelling blueprint for achieving the effective computational depth that contemporary artificial models lack. It organizes computation hierarchically across cortical regions operating at different timescales, enabling deep, multi-stage reasoning …
> 
> Inspired by this hierarchical and multi-timescale biological architecture, we propose the Hierarchical Reasoning Model (HRM). HRM is designed to significantly increase the effective computational depth. It features two coupled recurrent modules: a high-level (H) module for abstract, deliberate reasoning, and a low-level (L) module for fast, detailed computations
> 
> Figure 2: **The necessity of depth for complex reasoning**
> 
> Furthermore, we propose a one-step gradient approximation for training HRM, which offers improved efficiency and eliminates the requirement for BPTT. This design maintains a constant memory footprint (O(1) compared to BPTT’s O(T) for T timesteps) throughout the backpropagation process, making it scalable and more biologically plausible.  
> Leveraging its enhanced effective depth, HRM excels at tasks that demand extensive search and backtracking. **Using only 1,000 input-output examples, without pre-training or CoT supervision** , HRM learns to solve problems that are intractable for even the most advanced LLMs

> Figure 4: Top: Diagram of HRM with approximate gradient. Bottom: Pseudocode of HRM with deep supervision training in PyTorch.
> 
> **Adaptive computational time** (ACT) The brain dynamically alternates between automatic thinking (“System 1”) and deliberate reasoning (“System 2”) …  
> Inspired by the above mechanism, we incorporate an adaptive halting strategy into HRM that enables “thinking, fast and slow”. This integration leverages deep supervision and uses the Q-learning algorithm
> 
> **Inference-time scaling** … As illustrated in Figure 5-(c), HRM seamlessly achieves inference-time scaling
> 
> **Stability of Q-learning in ACT** The deep Q-learning that underpins our ACT mechanism is  
> known to be prone to instability, often requiring stabilization techniques such as replay buffers  
> and target networks, which are absent in our design. Our approach, however, achieves stability through the intrinsic properties of our model and training procedure. Recent theoretical work by Gallici et al. … Our model satisfies these conditions through its Post-Norm architecture that employs RMSNorm (a layer normalization variant) and the AdamW optimizer
> 
> **Architectural details** We employ a sequence-to-sequence architecture for HRM …  
> For all Transformer blocks in this work—including those in the baseline models—we incorporate the enhancements found in modern LLMs (based on Llama architectures). These improvements include Rotary Positional Encoding, Gated Linear Units, RMSNorm, and the removal of bias terms from linear layers.  
> …  
> Furthermore, both HRM and recurrent Transformer models implement a Post-Norm architecture with weights initialized via truncated LeCun Normal initialization, while the scale and bias  
> parameters are excluded from RMSNorm. All parameters are optimized using the Adam-atan2 optimizer, a scale-invariant variant of Adam  
> …  
> We introduce _Sudoku-Extreme_, a more challenging dataset that is compiled from the aforementioned easy datasets as well as puzzles recognized by the Sudoku community as exceptionally difficult for human players:  
> …  
> We use Sudoku-Extreme in our main experiments (Figure 1)
> 
> Remarkably, HRM attains these results with just ~1000 training examples per task—and  
> **without pretraining or CoT labels**
> 
> **3.3 Visualization of intermediate timesteps**  
> Although HRM demonstrates strong performance on complex reasoning tasks, it raises an intriguing question: what underlying reasoning algorithms does the HRM neural network actually implement? Addressing this question is important for enhancing model interpretability and developing a deeper understanding of the HRM solution space.
> 
> While a definitive answer lies beyond our current scope, we begin our investigation by analyzing  
> state trajectories and their corresponding solution evolution
> 
> **4 Brain Correspondence**  
> …  
> Figure 8: **Hierarchical Dimensionality Organization in the HRM and Mouse Cortex**.  
> …  
> The high-to-low PR ratio in HRM (zH /zL ≈ 2.98) closely matches that measured in the mouse  
> cortex (≈ 2.25). In contrast, conventional deep networks often exhibit neural collapse, where  
> last-layer features converge to a low-dimensional subspace. HRM therefore departs from the  
> collapse pattern and instead fosters a high-dimensional representation in its higher module
> 
> **5 Related Work**  
> …  
> Another notable work in this area is Recurrent Relational Networks (RRN)
> 
> **6 Discussions**  
> **Turing-completeness of HRM** Like earlier neural reasoning algorithms including the Universal  
> Transformer, HRM is computationally universal when given sufficient memory and time constraints. In other words, it falls into the category of models that can simulate any Turing machine, overcoming the computational limitations of standard Transformers discussed previously in the introduction. Given that earlier neural algorithm reasoners were trained as recurrent neural networks, they suffer from premature convergence and memory intensive BPTT. …

It references many intriguing papers e.g.:

Meta’s (paper updated in July 2025)

> **[Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized...](https://arxiv.org/abs/2410.09918)**
>
> In cognition theory, human thinking is governed by two systems: the fast and intuitive System 1 and the slower but more deliberative System 2. Analogously, Large Language Models (LLMs) can operate in two reasoning modes: outputting only the solutions...

> **Reinforcement learning for reasoning in large language models with one**  
> **training example** , 2025. URL [[2504.20571] Reinforcement Learning for Reasoning in Large Language Models with One Training Example](https://arxiv.org/abs/2504.20571)

[https://arxiv.org/pdf/2506.04761](https://arxiv.org/pdf/2506.04761)

> arXiv:2506.04761v2 [cs.LG] 25 Jun 2025  
> **Log-Linear Attention**

And this work/paper:

> **[GitHub - simplescaling/s1: s1: Simple test-time scaling](https://github.com/simplescaling/s1)**
>
> s1: Simple test-time scaling

Looking up post-norm I find:

> **[HybridNorm: A Hybrid Normalization Strategy Combining Pre-Norm and Post-Norm...](https://www.marktechpost.com/2025/03/12/hybridnorm-a-hybrid-normalization-strategy-combining-pre-norm-and-post-norm-strengths-in-transformer-architectures/)**
>
> HybridNorm: A Hybrid Normalization Strategy Combining Pre-Norm and Post-Norm Strengths in Transformer Architectures

> **[The Big LLM Architecture Comparison](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison)**
>
> From DeepSeek-V3 to Kimi K2: A Look At Modern LLM Architecture Design

[There are also 4 open source game AI generators out, seemingly as good as latest from Deepmind, which is still not released.]

I had in drafts from months ago (do not recall the exact context, believe this was Deepseek R1, when “reasoning” was new, and Julia code generated, for phase 1 of some neural compression proof-of-concept code I was doing):

> “Thought for 732 seconds”  
> [Outputted thinking process, 48 screenfuls! Something you can hide or see, unlike for OpenAI, and it at least started along the same lines as I was thinking, but yes, I did NOT read all of it, just confirmed the final code works.]
> 
> > So the code for compression seems to handle all cases. [..]

---

_[View the full topic](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796)._