# An LLM fine-tuned for Julia, call for comments + help

**URL:** <https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462>\
**Category:** Tooling\
**Tags:** ai, llm\
**Created:** [April 24, 2024, 3:33pm UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462 "2024-04-24T15:33:53Z")\
**Posts on this page:** 13\
**Page:** 2

<div class="post-metadata">

**Author:** ![cpfiffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cpfiffer/32/208747_2.png) [@cpfiffer](https://discourse.julialang.org/u/cpfiffer)\
**Post date:** [April 27, 2024, 1:31am UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/21 "2024-04-27T01:31:57Z")

</div>

Great, [here’s the PR](https://github.com/tylerjthomas9/JuliaCode/pull/1) to format all the code. This should give us a ~4x sample size, though for some of the formatters the differences are going to be quite small. I also don’t filter out unique data which maybe we want to do here.

An additional thing we could do on the formatting side is to make the permutations of [all the arguments](https://domluna.github.io/JuliaFormatter.jl/dev/api/#JuliaFormatter.format_text-Tuple%7BAbstractString%7D) to `format_text`:

```julia
format_text(
    text::AbstractString;
    style::AbstractStyle = DefaultStyle(),
    indent::Int = 4,
    margin::Int = 92,
    always_for_in::Union{Bool,Nothing} = false,
    for_in_replacement::String = "in",
    whitespace_typedefs::Bool = false,
    whitespace_ops_in_indices::Bool = false,
    remove_extra_newlines::Bool = false,
    import_to_using::Bool = false,
    pipe_to_function_call::Bool = false,
    short_to_long_function_def::Bool = false,
    long_to_short_function_def::Bool = false,
    always_use_return::Bool = false,
    whitespace_in_kwargs::Bool = true,
    annotate_untyped_fields_with_any::Bool = true,
    format_docstrings::Bool = false,
    align_struct_field::Bool = false,
    align_conditional::Bool = false,
    align_assignment::Bool = false,
    align_pair_arrow::Bool = false,
    conditional_to_if = false,
    normalize_line_endings = "auto",
    align_matrix::Bool = false,
    trailing_comma::Bool = false,
    trailing_zero::Bool = true,
    indent_submodule::Bool = false,
    separate_kwargs_with_semicolon::Bool = false,
    surround_whereop_typeparameters::Bool = true,
    variable_call_indent::Vector{String} = []
    short_circuit_to_if::Bool = false,
)::String

```

though I am not sure what people think about that.

---

<div class="post-metadata">

**Author:** ![cpfiffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cpfiffer/32/208747_2.png) [@cpfiffer](https://discourse.julialang.org/u/cpfiffer)\
**Post date:** [April 27, 2024, 1:32am UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/22 "2024-04-27T01:32:53Z")

</div>

Regarding fine tuning, it would be interesting to see if we could construct “backward prompts”, i.e ask a high-quality language model to give us _questions_ to generate a particular script or part of a script. Anyone have opinions there too?

---

<div class="post-metadata">

**Author:** ![ludgerpaehler](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ludgerpaehler/32/32672_2.png) [@ludgerpaehler](https://discourse.julialang.org/u/ludgerpaehler)\
**Post date:** [April 27, 2024, 5:53am UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/23 "2024-04-27T05:53:50Z")

</div>

Hey everyone,

Just picking this up from having seen @cpfiffer post on Twitter. While not being useful for any alignment training training with RLHF/DPO/KTO yet, we have already released a pretraining-scale dataset which includes Julia as _LLVM IR_ here are the paper, and the HuggingFace dataset:

- Paper: [[2309.15432] ComPile: A Large IR Dataset from Production Sources](https://arxiv.org/abs/2309.15432)
- Dataset: [llvm-ml/ComPile · Datasets at Hugging Face](https://huggingface.co/datasets/llvm-ml/ComPile)

You can pull just the subsection of it if you’d want to use it to seed efforts.

We are also going to release a version of Source Code → IR relatively soonish.

---

<div class="post-metadata">

**Author:** ![svilupp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/svilupp/32/34933_2.png) [@svilupp](https://discourse.julialang.org/u/svilupp)\
**Post date:** [April 27, 2024, 6:46am UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/24 "2024-04-27T06:46:47Z")

</div>

Yeah, that’s a common practice in creating evals. You can mimic the functionality in PromptingTools for that.

function: [Reference for RAGTools | PromptingTools.jl](https://svilupp.github.io/PromptingTools.jl/dev/reference_ragtools#PromptingTools.Experimental.RAGTools.build_qa_evals-Tuple%7BVector%7B%3C:AbstractString%7D,%20Vector%7B%3C:AbstractString%7D%7D)

template: [PromptingTools.jl](https://svilupp.github.io/PromptingTools.jl/dev/prompts/RAG#Template:-RAGCreateQAFromContext)

There is a blog on Forem on how to use it for RAG. The tweaking to get it to work on source code etc would be quite minimal.

btw a quick tip - JSON doesn’t have multiline strings even when pretty-formatted, so it’s painful to read and edit.  
BUT if you use VSCode (Cursor, …), you can install a few extensions to make it a breeze - “multiline string editor” ([Multiline String Editor - Visual Studio Marketplace](https://marketplace.visualstudio.com/items?itemName=chop-dbhi.multiline-string-editor)) and “JSON multiline viewer”([JSON multiline viewer - Visual Studio Marketplace](https://marketplace.visualstudio.com/items?itemName=AndreyOsiyuk.json-multiline-viewer)).

---

<div class="post-metadata">

**Author:** ![photor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/photor/32/14343_2.png) [@photor](https://discourse.julialang.org/u/photor)\
**Post date:** [April 29, 2024, 3:49am UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/25 "2024-04-29T03:49:10Z")

</div>

Awesome!

---

<div class="post-metadata">

**Author:** ![svilupp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/svilupp/32/34933_2.png) [@svilupp](https://discourse.julialang.org/u/svilupp)\
**Post date:** [April 29, 2024, 8:57am UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/26 "2024-04-29T08:57:52Z")

</div>

Btw if people are lazy auto-saving their conversations in REPL, try using this [GUI](https://discourse.julialang.org/t/ann-protoportal-jl-a-portal-to-the-efficient-coding-on-the-go-and-the-magic-of-promptingtools-jl/113575) for LLM questions. It saves all conversations by default when you click “New chat”.

Disclaimer: I’m the author of the tool, but I do believe it can help since it requires no extra effort.

In the next few days or so, I’ll also publish a simple observability platform for LLM conversations (saved in JSON), so we can quickly review/filter/curate the saved conversations.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [April 29, 2024, 4:54pm UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/27 "2024-04-29T16:54:50Z")

</div>

> [@ludgerpaehler](#):
>
> we have already released a pretraining-scale dataset which includes Julia as _LLVM IR_ here are the paper, and the HuggingFace dataset:
> 
> - Paper: [[2309.15432] ComPile: A Large IR Dataset from Production Sources](https://arxiv.org/abs/2309.15432)

That’s a very intriguing idea and paper. I’ve so far only scanned it, and it seems to me having register numbers in IR or assembly can obscure the meaning. I’m guessing LLVM IR was chosen since LLVM is a common backend.

Should `@code_lowered`, `@code_typed` down to `@code_native` be run on the Julia training code, and associated with it? Which, likely best, if not all? It seems problematic that Julia is generic, and there’s not just one set of types to run on, and you get different answers with different sets for each of those. I guess any one (or more) valid set could be ok.

I was also thinking of another language. I though stack-based Forth might be better (no register numbers), I’m not up-to-speed on much LLVM use for it, might not have lots of Forth code as training data, or users, since very old and people may not know of or care… but if you could compile Julia (or other language) to it then it might be good, and help the LLM see meaning in the (Julia) code.

I looked up what’s already being done with Forth, and I at least find some interesting papers:

A Neural Forth Abstract Machine

> **[rnn\_nampi.pdf](https://narad.github.io/papers/rnn_nampi.pdf)**
>
> 180.06 KB

NEURAL PROGRAMMER-INTERPRETERS  
[https://arxiv.org/pdf/1511.06279](https://arxiv.org/pdf/1511.06279)

---

<div class="post-metadata">

**Author:** ![mnemnion](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mnemnion/32/206596_2.png) [@mnemnion](https://discourse.julialang.org/u/mnemnion)\
**Post date:** [April 30, 2024, 2:48pm UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/28 "2024-04-30T14:48:21Z")

</div>

> [@Palli](#):
>
> One other thing about the data set, ours and what already trained on. Julia is a 1-based (and column-major) language, and I think we want more of that, Julia, in the training data but maybe ok to add R, MATLAB, Fortran etc. in our fine-tuning for that reason, but none that differ much, like Python, Rust, C/C++. [Also want code with e.g. `@view` in the training data.] We want the model to really understand the difference with Julia/1-based and Python/0-based and row-major NumPy.

I don’t know if or how the following observation is actionable in this project, but: we don’t want an LLM emitting code which assumes `arr[1]` is correct, it should be `arr[begin]` and so on, `zero(T)` instead of `0`.

The proper generic accessors are probably underutilized in a general Julia data set, it would be good to give code which does use them correctly some prominence.

---

<div class="post-metadata">

**Author:** ![cpfiffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cpfiffer/32/208747_2.png) [@cpfiffer](https://discourse.julialang.org/u/cpfiffer)\
**Post date:** [April 30, 2024, 3:29pm UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/29 "2024-04-30T15:29:23Z")

</div>

Ah this is a good point, we would also want to add stuff that the linter supports, such as using `for i in axes(thing)`. Anyone written anything that fixes these little things?

---

<div class="post-metadata">

**Author:** ![svilupp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/svilupp/32/34933_2.png) [@svilupp](https://discourse.julialang.org/u/svilupp)\
**Post date:** [May 20, 2024, 6:01am UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/30 "2024-05-20T06:01:55Z")

</div>

FYI.

From PromptingTools v0.26 onward you can achieve the auto-saving of your conversations by simply running this one line:

```julia
PT.register_model!(; name= "gpt-3.5-turbo", schema=PT.OpenAISchema() |> PT.TracerSchema |> PT.SaverSchema)

```

Apply for any model you use. It automatically logs necessary information (API kwargs, prompt templates and its versions, etc.) and then saves it to a disk (you can use LOG\_DIR env or provide your own path). For docs see `?SaverSchema`, `?TracerSchema`

---

<div class="post-metadata">

**Author:** ![cpfiffer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cpfiffer/32/208747_2.png) [@cpfiffer](https://discourse.julialang.org/u/cpfiffer)\
**Post date:** [May 20, 2024, 2:03pm UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/31 "2024-05-20T14:03:14Z")

</div>

Oh incredible! That is so cool. Love the wrapping approach.

---

<div class="post-metadata">

**Author:** ![Kirby\_Zhang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kirby_zhang/32/36204_2.png) [@Kirby\_Zhang](https://discourse.julialang.org/u/Kirby_Zhang)\
**Post date:** [July 16, 2025, 5:32pm UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/32 "2025-07-16T17:32:25Z")

</div>

It really should be metrics first. For the huge amount of effort to fine tune models that will be much smaller when deployed without cloud infrastructure, you have to be measuring something to motivate users.

---

<div class="post-metadata">

**Author:** ![adannenberg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adannenberg/32/7869_2.png) [@adannenberg](https://discourse.julialang.org/u/adannenberg)\
**Post date:** [February 2, 2026, 4:31pm UTC](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462/33 "2026-02-02T16:31:45Z")

</div>

this thread seems to have gone dormant… does anyone have updated opinions as to which llms / ides are “best” for julia programming now (feb 2026)? also, are the results in the llm leaderboard updated? i don’t see any datestamps so i have no idea when the rankings were generated (might have missed it…)

[Previous page](https://discourse.julialang.org/t/an-llm-fine-tuned-for-julia-call-for-comments-help/113462.md?page=1)