# An LLM fine-tuned for Julia, call for comments + help

**URL:** <https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462>\
**Category:** Tooling\
**Tags:** ai, llm\
**Created:** [April 24, 2024, 3:33pm UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462 "2024-04-24T15:33:53Z")\
**Posts on this page:** 1\
**Showing post:** 12

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [April 25, 2024, 11:19pm UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/12 "2024-04-25T23:19:34Z")

</div>

You need to choose at a minimum what model to finetune, and Lama3 is outdated already, and (I’ve only scanned the Slack thread to that point) this one, was good:

> Maybe this is useful ? [bigcode/starcoderbase · Hugging Face](https://huggingface.co/bigcode/starcoderbase)

> **[Snowflake/snowflake-arctic-instruct · Hugging Face](https://huggingface.co/Snowflake/snowflake-arctic-instruct)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Arctic LLM seems best now (the Base model updated 2 hours ago) and/or Phi-3 (for a small one, also new), would now be on my short-list (also **WaveCoder** and its paper also, for “LLM-based Generator-Discriminator data process framework to generate diverse, high-quality instruction data from open source code”, and also hybrid Mamba/Transformer [ajibawa-2023/Code-Jamba-v0.1 · Hugging Face](https://huggingface.co/ajibawa-2023/Code-Jamba-v0.1) still the only “Julia” tagged model on HF):

> [@AI tools to write (Julia) code (best/worse experience), e.g. ChatGPT, GPT 3.5](https://discourse.julialang.org/t/ai-tools-to-write-julia-code-best-worse-experience-e-g-chatgpt-gpt-3-5/91198/60):
>
> I feel obligated to state that state-space models (SSM), AND Transformers have provable limitations, for code related and e.g. chess, i.e. sequential, I can dig up the paper, e.g. also Mamba as affected as I recall, and then likely all Mamba/transformer hybrid such as Jamba, what I posted recently, but the paper has a solution, small fix to SSMs). Llama3 is brand-new but already outdated, even though it was best open source. More recent Phi-3 and Arctic are very intriguing (also Leave No Contex…

It’s a question is it better to start with a larger model (presumably better, but not always), or do they learn slower when fine-tuning? I.e. because of the size? You have a potential “catastrophic forgetting” with fine-tuning (for out-of-distribution data, so start with non-awful for Julia?), maybe not a bad thing forgetting the other languages Python etc…

One other thing maybe ruling out models is the tokenizer is fixed, some recent with about 30.000 possible tokens, and I’m thinking do they include the Unicode options Julia has? We want all the math operators supported/supportable…

---

_[View the full topic](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462)._