Hi,
New to Julia and trying to translate Julia code to Python and to Wolfram Mathematica and the other way around with ChatGPT however the result is buggy and requires too much overhead to deal with.
Any recommendation how to go about this? Or is the overhead inevitable?
Cheers
Rami
I often use Julia Hub’s AI engine or Blackbox in VS
There’s a tool somewhere to translate ("transpile/compile) Pythn to Julia somewhere. Though also consider just calling Python code from Julia, or in other direction with PythonCall.jl. It’s usually the best approach.
How you ask the models can matter, i.e. “Prompt Engineering”, but see new state-of-the-art (SOTA):
We tested AlphaCodium on a challenging code generation dataset called CodeContests, which includes competitive programming problems from platforms such as Codeforces. The proposed flow consistently and significantly improves results. On the validation set, for example, GPT-4 accuracy (pass@5) increased from 19% with a single well-designed direct prompt to 44% with the AlphaCodium flow. Many of the principles and best practices acquired in this work, we believe, are broadly applicable to general code generation tasks. Full implementation is available at: this https URL
I think AlphaCodium (brand new so I’m not fully sure yet) is not a model, but something you use with a (default) model, or alternative, to boost them, and then with this one would be my suggestion to try:
See:
stabilityai/stable-code-3b · Hugging Face
stable-code-3bis trained on 18 programming languages (selected based on the 2023 StackOverflow Developer Survey) and demonstrates state-of-the-art performance (compared to models of similar size) on the MultiPL-E metrics across multiple programming languages tested using BigCode’s Evaluation Harness.
What I found today. Nobody can probably tell you which open source (or not; paid for) is best. See also there what they compare to. You can run smaller models locally.
The best model is probably GPT 4, i.e. paid version of ChatGPT, but I’ve not used myself. I’m not sure, but what you get at JuliaHub is it (free access to the better paid version?) or at least GPT 3.5, which isn’t as good. Open source models are better than the free version, at least for other languages.
Probably no one is going to tell you which is best for Julia, because few test for it. Also few, train for it, e.g. not mentioned for that model, thou Julia IS in its benchmark,
Mistral company has very good models, e.g. mistral-medium, I hear, and (but unclear to me none of below are made for code, could try):
Also e.g.:
Inspired by DeepMind’s AlphaCode
better or already outdated, even in version 2 (version 1, previous SOTA, is outdated):
Note, if you look at benchmarks, even if a model is claimed best for coding, it may not be for Julia.
Julia is a “low resource” language according to Google PaLM’s technical report, and e.g. Python high-resource. It may have been in PaLM 2 report, but both are old, and Gemini now their best, and despite Python high-resource/highest fraction in most if not all models, and C++ high too, that doesn’t mean best for either. Despite that C++ is known to be poor in some, or all models, and Julia good, better than Python.
It could help if Julia had a larger fraction of the training data, but it doesn’t seem to matter too much, likely because similar to Python, and being a simple language. Humans and AI apparently learn Julia/simpler languages faster, as opposed to e.g. C++.
[Given that Julia is low-resource, few percent, or was it 0.x% Mathematica is pretty close to be near zero. Though I haven’t confirmed, but don’t recall seeing it mention, or as a fraction, or even as part of AI coding benchmarks. I’m curious how well it’s handled already…]
It probably matters if Julia has a 0% fraction of the training data, none or very low.
From the awesome Julia AI list:
- ReplGPT.jl - Brings ChatGPT interface as a Julia REPL mode.
- HelpGPT.jl - Calls ChatGPT to explain any errors in Julia code.
- GenGPT3.jl - A Gen.jl generative function that wraps the OpenAI API.
- GPTCodingTools.jl - Code generation tool for Julia language with useful prompt templates and self-healing features (ala OpenAI Code Interpreter). It does work, but development has been abandoned. (Disclaimer: I’m the author of this package.)
- PromptingTools.jl - Helps with everyday applications of Large Language Models in Julia by wrapping coming APIs, re-using prompts via templates, and enabling easy transition between different model providers (eg, OpenAI, Ollama). (Disclaimer: I’m the author of this package.)
- LLMTextAnalysis.jl - Leverage Large Language Models to uncover, evaluate, and label themes/concepts/spectra in large document collections.
Some of the tools there maybe be outdated because of better solutions I already pointed to.
Finally found the thread I was looking for:
Known best for Julia (but most likely outdated and needs to be updated):
Brand new, not using latest AI models I just posted (but could?):