# Run the Gemma 2 9B LLM in Julia and without a GPU?

**URL:** <https://discourse.julialang.org/t/run-the-gemma-2-9b-llm-in-julia-and-without-a-gpu/124155>\
**Category:** Machine Learning\
**Tags:** kaggle, llm\
**Created:** [December 24, 2024, 1:45pm UTC](https://discourse.julialang.org/t/run-the-gemma-2-9b-llm-in-julia-and-without-a-gpu/124155 "2024-12-24T13:45:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [December 24, 2024, 1:45pm UTC](https://discourse.julialang.org/t/run-the-gemma-2-9b-llm-in-julia-and-without-a-gpu/124155/1 "2024-12-24T13:45:24Z")

</div>

I’m ignorant about LLMs, machine learning, etc.; but I want to run the Gemma 2 9B large language model. I’m not interested in using it as a chat bot, rather the reason I want to use it is the fact that it defines the evaluation metric for Kaggle’s Santa 2024 optimization heuristic competition. I wonder if it’s possible to run the Gemma 2 9B LLM:

- In pure Julia
- On a CPU, without relying on a GPU. I know this would be much slower, if possible at all, but I feel like stepping through a debugger and using tools like Cthulhu.jl could help give me a better understanding of how the LLM works.

Gemma 2 9B links:

- [Google | Gemma 2 | Kaggle](https://www.kaggle.com/models/google/gemma-2/transformers/gemma-2-9b)
- [google/gemma-2-9b · Hugging Face](https://huggingface.co/google/gemma-2-9b)

Santa 2024:

- [Santa 2024 - The Perplexity Permutation Puzzle | Kaggle](https://www.kaggle.com/competitions/santa-2024/)

---

<div class="post-metadata">

**Author:** ![mrufsvold](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mrufsvold/32/31600_2.png) [@mrufsvold](https://discourse.julialang.org/u/mrufsvold)\
**Post date:** [December 24, 2024, 7:00pm UTC](https://discourse.julialang.org/t/run-the-gemma-2-9b-llm-in-julia-and-without-a-gpu/124155/2 "2024-12-24T19:00:11Z")

</div>

I too am ignorant here, so hopefully someone else will weigh in, but what made you settle on that model?

I saw someone posting about getting Llama models working in pure Julia recently (I’m on my phone, but I could track it down later). Would those work?

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [December 24, 2024, 7:49pm UTC](https://discourse.julialang.org/t/run-the-gemma-2-9b-llm-in-julia-and-without-a-gpu/124155/3 "2024-12-24T19:49:28Z")

</div>

> [@mrufsvold](#):
>
> what made you settle on that model?

As I tried to explain above in short, the Santa 2024 competition result submissions are all evaluated with that specific LLM, in a well-defined manner. So it’s not my choice, and Llama won’t work, only Gemma 2 9B specifically.

---

<div class="post-metadata">

**Author:** ![findmyway](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/findmyway/32/4946_2.png) [@findmyway](https://discourse.julialang.org/u/findmyway)\
**Post date:** [December 26, 2024, 8:53am UTC](https://discourse.julialang.org/t/run-the-gemma-2-9b-llm-in-julia-and-without-a-gpu/124155/4 "2024-12-26T08:53:36Z")

</div>

> [@nsajko](#):
>
> I wonder if it’s possible to run the Gemma 2 9B LLM:
> 
> - In pure Julia
> - On a CPU, without relying on a GPU. I know this would be much slower, if possible at all, but I feel like stepping through a debugger and using tools like [Cthulhu.jl](https://juliaregistries.github.io/General/packages/redirect_to_repo/Cthulhu) could help give me a better understanding of how the LLM works.

That’s possible.

TBH, I’m still a little confused about your purpose here. If you were to win the competition, I’d suggest calculating the perplexity in python code (with `PythonCall.jl` maybe) to avoid any potential inconsistency. But if you were to understand how the Gemma 2 model works, you’d better start with the llama models first. If you look into the model architecture, you’ll find that there are only very small differences.

I’m working on a tutorial to explain how to do fast LLM inference in pure Julia, but it has not yet been finished. For now, you may take a look at the following code snippets.

- [gpt2](https://github.com/JuliaGenAI/Master-Deep-Generative-Models-in-Julia/blob/main/05_Transformer/gpt2.jl)
- [llama3](https://github.com/JuliaGenAI/Master-Deep-Generative-Models-in-Julia/blob/main/05_Transformer/llama3.jl)

[Source](https://developers.googleblog.com/en/gemma-explained-new-in-gemma-2/)

 ![](https://global.discourse-cdn.com/julialang/original/3X/b/b/bbe806f6b95093ac274e640a84a1925e8e89692c.png)

---

<div class="post-metadata">

**Author:** ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)\
**Post date:** [December 26, 2024, 9:37am UTC](https://discourse.julialang.org/t/run-the-gemma-2-9b-llm-in-julia-and-without-a-gpu/124155/5 "2024-12-26T09:37:49Z")

</div>

> [@findmyway](#):
>
> TBH, I’m still a little confused about your purpose here.

My hope was that examining the LLM as a white box might allow either simplifying the evaluator while retaining accuracy, or figuring out some upper or lower bounds on it. However I’m beginning to think I was overly optimistic in this regard 😅

---

<div class="post-metadata">

**Author:** ![findmyway](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/findmyway/32/4946_2.png) [@findmyway](https://discourse.julialang.org/u/findmyway)\
**Post date:** [April 2, 2025, 11:01am UTC](https://discourse.julialang.org/t/run-the-gemma-2-9b-llm-in-julia-and-without-a-gpu/124155/6 "2025-04-02T11:01:37Z")

</div>

Gemma3 implemented in pure Julia without GPU 😉 :

> <https://github.com/JuliaGenAI/Master-Deep-Generative-Models-in-Julia/blob/main/04_gemma/gemma3.jl>