Hi all,
We’re setting up a self-hosted LLM for coding, and Julia is our primary language. The usual benchmarks (SWE-bench, Aider) barely cover Julia, so published scores don’t help much.
I assume that a lot of the fellows here are using such models and I’m not hte first one, so it’d be great to hear what your’e using and why!
Best,
Niv
hi, just a couple days ago there was discussion in this thread which seems pretty much what you’re interested in
Thank you! I’ll have a look now
My theory is that if an LLM is too dumb to abstract across the relatively minor differences between synatx and semantics in different programming languages, it’s going to be completely hopeless at abstracting across the various application domains that these languages are used in.
Julia code is just much much more similar to C++ or Python code than writing a Web server is to writing an aereodynamical simulation.
I wouldn’t pay too much attention to language specific benchmarks, and look at more general programming performance.