# \[ANN\] Julia LLM Leaderboard - Help us make it more relevant for every day problems!

**URL:** <https://discourse.julialang.org/t/ann-julia-llm-leaderboard-help-us-make-it-more-relevant-for-every-day-problems/107319>\
**Category:** Package Announcements\
**Tags:** announcement, ai, prompting\
**Created:** [December 8, 2023, 9:25am UTC](https://discourse.julialang.org/t/ann-julia-llm-leaderboard-help-us-make-it-more-relevant-for-every-day-problems/107319 "2023-12-08T09:25:04Z")\
**Posts on this page:** 3\
**Page:** 2

<div class="post-metadata">

**Author:** ![svilupp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/svilupp/32/34933_2.png) [@svilupp](https://discourse.julialang.org/u/svilupp)\
**Post date:** [February 23, 2024, 6:11pm UTC](https://discourse.julialang.org/t/ann-julia-llm-leaderboard-help-us-make-it-more-relevant-for-every-day-problems/107319/21 "2024-02-23T18:11:35Z")

</div>

I can’t wait to try this one: [Phind](https://www.phind.com/blog/introducing-phind-70b)

---

<div class="post-metadata">

**Author:** ![alfaromartino](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alfaromartino/32/52986_2.png) [@alfaromartino](https://discourse.julialang.org/u/alfaromartino)\
**Post date:** [February 23, 2024, 6:51pm UTC](https://discourse.julialang.org/t/ann-julia-llm-leaderboard-help-us-make-it-more-relevant-for-every-day-problems/107319/22 "2024-02-23T18:51:42Z")

</div>

Phind didn’t pass my usual test (so far, actually no LLM has been able to do so). I always ask the question “in Julia, do slices behave as copies or views?”. It was giving the wrong answer.

---

<div class="post-metadata">

**Author:** ![svilupp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/svilupp/32/34933_2.png) [@svilupp](https://discourse.julialang.org/u/svilupp)\
**Post date:** [April 5, 2024, 8:48pm UTC](https://discourse.julialang.org/t/ann-julia-llm-leaderboard-help-us-make-it-more-relevant-for-every-day-problems/107319/23 "2024-04-05T20:48:17Z")

</div>

It’s not yet a full release (there is a bit of work left still), but I wanted to share some preliminary findings about the new Anthropic models.

Turns out Claude-3 Haiku broke the value-for-money barrier previously owned by GPT-3.5-Turbo:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/d/9/d986c5d557f32a5aca166d865e244866461f1461.png)

I’ve been really impressed by Claude Opus but the real star of the show here is Claude Haiku!  
I paid $30 for the Opus evals, but only **$0.4 for Haiku** evals!!!  
(and Opus had a crazy bad availability, I had to restart many times)

Btw. you can now use Anthropic models also for data extraction/function calling (released today) - it’s available in v0.18 of PromptingTools among [other things](https://forem.julialang.org/svilupp/the-latest-scoop-on-promptingtoolsjl-n2n).

What’s next? Exciting GSoC project (hopefully), new test cases, new models, and a new category of evals… A lot to look forward to.

[Previous page](https://discourse.julialang.org/t/ann-julia-llm-leaderboard-help-us-make-it-more-relevant-for-every-day-problems/107319.md?page=1)