I just discovered that ArtificialAnalysis (major independent LLM benchmarking org) actually has a Julia-specific benchmark now, so I had to post it here:
This benchmark measures language knowledge, NOT coding skill. It’s basically a Q&A test, with hallucinations deducted from correct answers (so the score ranges from -100 to +100).
Claude obliterates the field here, especially in comparison with GPT. Apparently Anthropic makes a dedicated effort to train on Julia code (though Opus also leads in general knowledge, see below).
The peak score for Opus (80) is quite decent, on par with Python’s (79).
But the average scores for these top models are much higher for Python than for Julia (switch the dropdown in the live chart linked below).
I also included the top open weights models from China for comparison. They are near SOTA in intelligence, though they lag a bit in knowledge - especially niche stuff like Julia.
Despite my disclaimer in bullet 1 above, I suspect that this knowledge advantage for Claude also translates to a real advantage when coding in Julia, but unfortunately there are no other language specific benchmarks to confirm this notion. At the very least, I would expect more idiomatic Julia code from Claude than from competitors.
Interesting stuff. I wonder if the reason that Claude does so well on Julia code comes down to the fact that the julia language devs mostly use Claude, which means that there’s a lot of high-quality examples of a developer instructing Claude to solve julia problems, which then ends up contributing a lot to subsequent training runs.
Just because an agent doesn’t have the knowledge doesn’t mean it can’t discover what it needs through exploration/trial and error. It’s just far less context efficient, so it helps to dispatch a separate/sub-agent to explore, design, stress/refine, and then have the supervisor implement the design within its own clean context window. For day to day tasks I find this works well enough in practice that I don’t need Claude for most things. You can also call Claude Code in non-interactive mode for the most complicated tasks and design work from other agents. This works with even the cheapest Claude subscription.
That benchmark is weird… Gemma 4 E4B and E2B are leading the small open source models for Julia. These models are cool for their size, I tried using E4B on my phone today, giving it a photo with a table of dates and the instruction “add it to my calendar” and it got most of them wrong (took around 15 minutes)… I am happy to see that Facebook seems to have a good open source model! And what is g9v3?
