# \[Help Wanted\] Help contribute test cases to improve LLM performance on Julia code

**URL:** <https://discourse.julialang.org/t/help-wanted-help-contribute-test-cases-to-improve-llm-performance-on-julia-code/132991>\
**Category:** Community\
**Tags:** llm\
**Created:** [October 8, 2025, 6:38pm UTC](https://discourse.julialang.org/t/help-wanted-help-contribute-test-cases-to-improve-llm-performance-on-julia-code/132991 "2025-10-08T18:38:43Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Keno](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keno/32/285_2.png) [@Keno](https://discourse.julialang.org/u/Keno)\
**Post date:** [October 8, 2025, 6:38pm UTC](https://discourse.julialang.org/t/help-wanted-help-contribute-test-cases-to-improve-llm-performance-on-julia-code/132991/1 "2025-10-08T18:38:44Z")

</div>

Hi all,

For the past few weeks, I’ve been working on creating a set of benchmark test cases that will be used to evaluate and train LLMs to improve their performance on Julia code. I’m particularly interested in test cases that people have tried to use AI agents on, but they barely failed or appear to be just beyond the capability frontier of current leading-edge agents. This’ll all be open source in the medium term future, but at the moment I’m keeping it a little smaller to make sure I can help people get the test cases write and provide API credits to get pass rates, etc. If you’re interested in participating, please ping me on slack.

Thanks!

---

<div class="post-metadata">

**Author:** ![greatpet](https://avatars.discourse-cdn.com/v4/letter/g/e495f1/32.png) [@greatpet](https://discourse.julialang.org/u/greatpet)\
**Post date:** [October 9, 2025, 9:52am UTC](https://discourse.julialang.org/t/help-wanted-help-contribute-test-cases-to-improve-llm-performance-on-julia-code/132991/2 "2025-10-09T09:52:34Z")

</div>

LLMs tend to write slightly outdated code when it comes to Flux.jl, as the package API changed a bit over the last few years. GPT-5 gives me code that declares custom layers with `@functor` rather than `@layer`, but the latter is recommended in recent versions of Flux.jl. Generally speaking, fine-tuning LLMs to write “modern” Julia may be useful.

---

<div class="post-metadata">

**Author:** ![Keno](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keno/32/285_2.png) [@Keno](https://discourse.julialang.org/u/Keno)\
**Post date:** [October 9, 2025, 11:54am UTC](https://discourse.julialang.org/t/help-wanted-help-contribute-test-cases-to-improve-llm-performance-on-julia-code/132991/3 "2025-10-09T11:54:55Z")

</div>

Write me a test case for it and that’ll happen :).

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [October 9, 2025, 12:09pm UTC](https://discourse.julialang.org/t/help-wanted-help-contribute-test-cases-to-improve-llm-performance-on-julia-code/132991/4 "2025-10-09T12:09:27Z")

</div>

Is gives bad code for DataFramesMeta.jl , it ends up being some weird mis-mash of DataFramesMeta and dplyr.

---

<div class="post-metadata">

**Author:** ![greatpet](https://avatars.discourse-cdn.com/v4/letter/g/e495f1/32.png) [@greatpet](https://discourse.julialang.org/u/greatpet)\
**Post date:** [October 10, 2025, 10:17am UTC](https://discourse.julialang.org/t/help-wanted-help-contribute-test-cases-to-improve-llm-performance-on-julia-code/132991/5 "2025-10-10T10:17:52Z")

</div>

Prompt to GPT-5:

> Write a minimal example for defining a custom layer in Flux.jl. Only give code not explanations.

The response was (WARNING: invalid code below)

```julia-auto
using Flux

struct MyLayer
    W::Matrix{Float32}
    b::Vector{Float32}
end

Flux.@functor MyLayer

MyLayer(in, out) = MyLayer(Flux.glorot_uniform(out, in), zeros(Float32, out))

(m::MyLayer)(x) = m.W * x .+ m.b

m = MyLayer(3, 2)
x = rand(Float32, 3, 5)
y = m(x)

gs = gradient(params(m)) do
    sum(abs2, m(x))
end

```

The code has a minor error, `params` instead of `Flux.params`, and running the corrected code triggers 3 deprecation warnings from Flux. My attempt at a modernized version is

```julia-auto
using Flux

struct MyLayer
    W::Matrix{Float32}
    b::Vector{Float32}
end

Flux.@layer MyLayer

MyLayer(in, out) = MyLayer(Flux.glorot_uniform(out, in), zeros(Float32, out))

(m::MyLayer)(x) = m.W * x .+ m.b

m = MyLayer(3, 2)
x = rand(Float32, 3, 5)
y = m(x)

gs = Flux.withgradient(m) do model
    sum(abs2, model(x))
end

```

---

<div class="post-metadata">

**Author:** ![Keno](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keno/32/285_2.png) [@Keno](https://discourse.julialang.org/u/Keno)\
**Post date:** [October 23, 2025, 6:06pm UTC](https://discourse.julialang.org/t/help-wanted-help-contribute-test-cases-to-improve-llm-performance-on-julia-code/132991/6 "2025-10-23T18:06:29Z")

</div>

This project is now public at [GitHub - JuliaBench/JuliaBench: LLM Benchmark problems for SWE tasks in julia](https://github.com/JuliaBench/JuliaBench) - please feel free to submit PRs even with WIP problem definitions. I can help get the graders working and the pass rates tuned.

---

<div class="post-metadata">

**Author:** ![sairus7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sairus7/32/10816_2.png) [@sairus7](https://discourse.julialang.org/u/sairus7)\
**Post date:** [November 10, 2025, 1:56pm UTC](https://discourse.julialang.org/t/help-wanted-help-contribute-test-cases-to-improve-llm-performance-on-julia-code/132991/7 "2025-11-10T13:56:09Z")

</div>

Copilot tends to paste boilerplate from well-knows libs that are already wrapped by Julia packages. E.g. when I’m using [GitHub - JuliaImGui/CImGui.jl: Julia wrapper for cimgui](https://github.com/JuliaImGui/CImGui.jl) it suggest to write full rendering code again.