# My journey training an LLM from scratch in Julia (and why I see huge potential)

**URL:** <https://discourse.julialang.org/t/my-journey-training-an-llm-from-scratch-in-julia-and-why-i-see-huge-potential/136843>\
**Category:** General Usage\
**Tags:** package, announcement, plotting, flux, dataframes\
**Created:** [April 23, 2026, 11:32pm UTC](https://discourse.julialang.org/t/my-journey-training-an-llm-from-scratch-in-julia-and-why-i-see-huge-potential/136843 "2026-04-23T23:32:15Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![mouad-tarif](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mouad-tarif/32/221676_2.png) [@mouad-tarif](https://discourse.julialang.org/u/mouad-tarif)\
**Post date:** [April 23, 2026, 11:32pm UTC](https://discourse.julialang.org/t/my-journey-training-an-llm-from-scratch-in-julia-and-why-i-see-huge-potential/136843/1 "2026-04-23T23:32:15Z")

</div>

I started training a language model from scratch in Julia. Not using pre-built libraries for the core - building my own BPE tokenizer, my own training loop, facing hallucinations, and rebuilding.

I tried Flux and Lux. Both have strengths, but also critical weaknesses (CUDA conflicts, design limitations). After a long struggle, I found a different path that worked.

PythonCall played a key role, bridging Julia to Python’s ecosystem when needed. But Julia itself was the heart of the project.

What I learned is that Julia is not just “another language”. It is a platform for real understanding. If the community focuses on its unique strengths (speed, metaprogramming, Python interop), I believe Julia can surpass many expectations.

I am not sharing technical details now. But I wanted to confirm: Julia is inspiring. With more work, it can become much, much more.  
“I am sharing a screenshot as a proof of concept. The full code is not open-source at this stage. I want to document it properly first. I may share it later. I hope you understand and respect that.”  
#machinelearning

> 

 ![Capture d’écran du 2026-04-23 22-11-01](https://global.discourse-cdn.com/julialang/original/3X/5/4/54b6e74c3a5bd9ae756baa4d950c23555766469a.png)

---

<div class="post-metadata">

**Author:** ![lilachint](https://avatars.discourse-cdn.com/v4/letter/l/ee7513/32.png) [@lilachint](https://discourse.julialang.org/u/lilachint)\
**Post date:** [April 24, 2026, 5:52am UTC](https://discourse.julialang.org/t/my-journey-training-an-llm-from-scratch-in-julia-and-why-i-see-huge-potential/136843/2 "2026-04-24T05:52:18Z")

</div>

When I first came across Julia a few weeks ago, it really is my dream come true! So many language features and design that I adore (I once tried designing an intuitive language myself, and turns out Julia is literally what I was looking for the whole time). I could confirm Julia is so inspiring, and in my opinion at least, Julia is THE language (well, Julia is unironically a cult lol).

---

<div class="post-metadata">

**Author:** ![davidbp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidbp/32/463_2.png) [@davidbp](https://discourse.julialang.org/u/davidbp)\
**Post date:** [April 24, 2026, 6:54am UTC](https://discourse.julialang.org/t/my-journey-training-an-llm-from-scratch-in-julia-and-why-i-see-huge-potential/136843/3 "2026-04-24T06:54:25Z")

</div>

Please consider sharing the code, would be really interested taking a look. This discourse is one of the best ways to learn and share julia tips and tricks.

I would be specially interested looking at the BPE tokenizer from scratch.

---

<div class="post-metadata">

**Author:** ![mouad-tarif](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mouad-tarif/32/221676_2.png) [@mouad-tarif](https://discourse.julialang.org/u/mouad-tarif)\
**Post date:** [April 24, 2026, 12:04pm UTC](https://discourse.julialang.org/t/my-journey-training-an-llm-from-scratch-in-julia-and-why-i-see-huge-potential/136843/4 "2026-04-24T12:04:32Z")

</div>

Thank you for your interest. I understand that the Discourse is for learning, and I appreciate that.

However, as I mentioned in the original post, the code is not open-source at this stage. I am still documenting it and refining it.

Regarding the BPE tokenizer: I built it from scratch after studying multiple implementations. The general approach is standard (Byte-Pair Encoding), but my specific adaptation for Arabic text and the model’s vocabulary is what makes it unique.

I am not sharing the code yet, but I am happy to discuss the algorithm or the challenges I faced. What specifically would you like to know about BPE in Julia?

---

<div class="post-metadata">

**Author:** ![mouad-tarif](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mouad-tarif/32/221676_2.png) [@mouad-tarif](https://discourse.julialang.org/u/mouad-tarif)\
**Post date:** [April 24, 2026, 2:03pm UTC](https://discourse.julialang.org/t/my-journey-training-an-llm-from-scratch-in-julia-and-why-i-see-huge-potential/136843/5 "2026-04-24T14:03:43Z")

</div>

confimation using julia

 ![Capture d’écran du 2026-04-24 00-41-28](https://global.discourse-cdn.com/julialang/original/3X/4/e/4e343cb3ac9f48db8626883c9fee23a3c092591b.png)

---

<div class="post-metadata">

**Author:** ![mouad-tarif](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mouad-tarif/32/221676_2.png) [@mouad-tarif](https://discourse.julialang.org/u/mouad-tarif)\
**Post date:** [April 24, 2026, 5:07pm UTC](https://discourse.julialang.org/t/my-journey-training-an-llm-from-scratch-in-julia-and-why-i-see-huge-potential/136843/6 "2026-04-24T17:07:59Z")

</div>

good luck in your dream