I have tested dozens of combinations of agents, quantisations, ‘speedup tricks’, etc. over past several months on Julia coding (specifically for mathematical programming and statistical learning) with dismal results, until the past few days, when I discovered Agent_A1 and spent time tuning it. The results are incredible - 1-shot solutions to challenging test sets for the first time ever!
[albeit on my own idiosyncratic choice of tests]
For a full rundown, see an article I wrote on Medium (I will post elsewhere if I can figure out where): https://medium.com/@compleathorseplayer/i-spent-weeks-hunting-for-a-local-llm-that-can-actually-code-for-statistical-learning-julia-or-e18f7f269217
[note: my experience is on a Mid-size Macbook, but I hope my experience is enlightening]
TLDR: The metrics driving local coding development tend to be for ‘business-orientated’ programming tasks rather than Scientific projects, with predictable gaps. This new ‘model of models’ seems to address this (for the first time I have seen in the non-huge Local Models context)
Very nice! Any idea if it will also run on Linux with a graphics card with large memory?
Great take, and indeed enlightening. Among the populars, I find Claude the most useful, albeit for less challenging work than this. But it has an unfortunate tendency to dig itself into holes, probably due to headroom squeeze when context expands to fill the room available for it. It will announce successive failures with “here it the final, final fix.” At that point, the only thing to get unstuck is to take the failed attempt over to Gemini or Codex.
@ufechner7, the Silicon memory is unified and that means the internal tossing of the ball back and forth from CPU to GPU is far, far faster than you can expect even on a well-provisioned Linux, as I found in my brief experience with NVIDIA Spark DGX. (I returned it for an intractable NIC problem that limited LAN transfers to the fallback of 200MB on what was supposed to be a 10GB port. No one ever admitted that it was due to troubles with the Realtek driver, but they did give me my money back after the exchange period.) Long way round to “should work” if you can get around the Flux issues (Linux won’t have the MLX) and are. willing to wait on the handoffs.
I’ll take a look at that model, sounds interesting.
I tried your Rosenbrock benchmark task using an external LLM within my soon to be released platform KaimonSlate, which builds upon Kaimon.jl and extends that tight AI pairing to a cell based notebook system which also has features for export to typeset documents and web platforms, GH pages, etc. You can use it with local models as well, through Ollama and vMLX.
What kind of results are you getting with the Agent_A1?