# Community Interest Check: LLMs from Scratch in Pure Julia

**URL:** <https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796>\
**Category:** Offtopic\
**Tags:** package\
**Created:** [October 26, 2024, 7:10pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796 "2024-10-26T19:10:38Z")\
**Posts on this page:** 1\
**Showing post:** 26

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [November 29, 2024, 12:16pm UTC](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796/26 "2024-11-29T12:16:15Z")

</div>

> [@mantzaris](#):
>
> " .. _But even Julia’s Float16 might be ok for KANs”_ I was not aware that this new trend had such an impact and the direction was totally changed. How much of the community uses KANs now?

I was talking out bfloat16 being dominant for training, not Float32, and then quantized used a lot. But not KANs, they are new. Sorry for the confusion, I just immediately continued with mentioning them, didn’t explicitly say they were popular yet, but I think they will be; in transformers. Quantization _to 4-bit_ is I think mainstream, though I often see NONE used, i.e. models released first with e.g. bloat16, and then the quantization community post-quantizees models, and maybe fine-tunes.

I see from @ForceBru “Just use automatic differentiation” for backpropagation for KANs. And in a paper I linked “trained four KAN models using PyTorch, each sized 17x1x14, with G values of 7, 15, 30, and 60 corresponding to array sizes of 128, 256, 512, and 1024, respectively.” So I certainly think KANs would fit into Flux.jl. While I’m no expert on Flux or Lux, if KANs do not fit there then they should. An alternative, and an ok first step is to do independently of as with: “we use KANs as a nice opportunity to implement them from scratch in simple Python (no PyTorch / TensorFlow: just some good old numpy!).”

Amazing page:

> **[{model\_size, hyperparameters, - Training LLMs at Scale](https://samforeman.me/talks/llms-at-scale/#large-language-models)**
>
> Training LLMs at Scale

Look at e.g.

> ### Parallelism Concepts

And Julia is most likely behind.

It’s not too important to do everything from scratch (see e.g. Jjama3.jl, being impure, @noob, @AntonOresten, depending on Python/Rust code, but why not Rust directly? It’s other BytePairEncoding.jl is though a"Pure Julia implementation of the Byte Pair Encoding (BPE) method."):

> [@\[ANN\] Jjama3.jl (unregistered) - Llama3.1 and Llama3.2 (text) in Julia](https://discourse.julialang.org/t/ann-jjama3-jl-unregistered-llama3-1-and-llama3-2-text-in-julia/122641/4):
>
> - Life is too short to spend on tokenizers, so we switched to using [a julia wrapper of a python wrapper of Hugging Face’s rust tokenizers library](https://github.com/MurrellGroup/HuggingFaceTokenizers.jl) (hat tip @AntonOresten for this!).
> - Added low-rank (LoRA) finetuning.
> - Added new samplers, including classics top-p & top-k [..]
> 
> Switching tokenizers unlocked some fun small open models, like the SmolLM2 series (more open than Llama3.2, which is behind a permissions wall, so this might reduce a barrier to getting started). With the LoRA addition, this is at a fairly decent point for someone wanting to tinker with LLMs.

I absolutely agree we _shouldn’t_ bother implementing tokenizers in Julia, rather reuse, and **even better get rid of** (I also see Karpathy is now at a new AI company after, Eureka Labs, after leaving OpenAI, and Tesla before):

> <https://x.com/karpathy/status/1657949234535211009>
>
> AK @\_akhaliq

> There is a whole separate stage with its own training and inference, and additional libraries. It complicates the ingest of additional modalities. Tokenization also has many subtle sharp edges. Few examples: [..]  
> Tokenization creates attack surfaces, e.g. SolidGoldMagikarp [..]  
> The list goes on, TLDR everyone should hope that tokenization could be thrown away. Maybe even more importantly, we may find general-purpose strategies for multi-scale training in the process.

Looking into “multi-scale training” I find a lot (most not directly on LLMs, on images, or time-series, not sure if the ideas translate to LLMs):  
[https://medium.com/visionwizard/sniper-efficient-multi-scale-training-826e90573b20](https://medium.com/visionwizard/sniper-efficient-multi-scale-training-826e90573b20)

> **[LLM-Mixer: Multiscale Mixing in LLMs for Time Series Forecasting](https://arxiv.org/html/2410.11674)**

> **[Multiscale Positive-Unlabeled Detection of AI-Generated Texts](https://arxiv.org/abs/2305.18149)**
>
> Recent releases of Large Language Models (LLMs), e.g. ChatGPT, are astonishing at generating human-like texts, but they may impact the authenticity of texts. Previous works proposed methods to detect these AI-generated texts, including simple ML...

See also there open-sora:

> **[Release MindONE 0.2.0 · mindspore-lab/mindone](https://github.com/mindspore-lab/mindone/releases/tag/v0.2.0)**
>
> We are excited to announce the official release of MindONE, a state-of-the-art repository dedicated to multi-modal understanding and content generation. Built on MindSpore 2.3.1 and optimized for ...

> <https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/docs/Report-v1.3.0.md>

[https://github.com/VQAssessment/DOVER](https://github.com/VQAssessment/DOVER)

Something I was seeing but not looked at close enough to know if relevant for us:

This seems important and only 9 pages:  
[https://arxiv.org/pdf/2407.00952](https://arxiv.org/pdf/2407.00952)

[https://arxiv.org/pdf/2405.09394](https://arxiv.org/pdf/2405.09394)

> Experimental results demonstrate that SA-FedLoRA is an efficient FL, achieving superior performance to FedAvg and significantly reducing communication parameters by up to 93.62%"

> **[FedLoRA: When Personalized Federated Learning Meets Low-Rank...](https://openreview.net/forum?id=bZh06ptG9r)**
>
> In this research paper, we introduce a novel approach to Personalized Federated Learning (PFL), which we call FedLoRA. This approach is inspired by recent advancements in fine-tuning Large Language...

I did not expect to see “wireless” and “jamming resistant” in relation to LLMs:

### R-SFLLM: Jamming Resilient Framework for Split Federated Learning with Large Language Models

[https://arxiv.org/pdf/2407.11654](https://arxiv.org/pdf/2407.11654)

[https://medium.com/@plienhar/llm-inference-series-4-kv-caching-a-deeper-look-4ba9a77746c8](https://medium.com/@plienhar/llm-inference-series-4-kv-caching-a-deeper-look-4ba9a77746c8)

> **[How do we measure the ROI of generative AI?](https://cohere.com/blog/how-do-we-measure-the-roi-of-generative-ai-2)**
>
> From cost uncertainties to unclear benefits, measuring GenAI’s true value can be tough, but not impossible.

> **[Reddit - The heart of the internet](https://www.reddit.com/r/MachineLearning/comments/1cfj9kf/crosspost_on_improving_llm_efficiency_using_split/)**

---

_[View the full topic](https://discourse.julialang.org/t/community-interest-check-llms-from-scratch-in-pure-julia/121796)._