Hey y’all! @tbeason suggested that I put up a little thread asking people about what misc AI tooling they actually use. I mostly just want to understand how people are using these tools in the ecosystem.
First – a note on general language model performance in Julia. @svilupp has a fantastic leaderboard for which LLMs tend to do better. At the moment Claude 3 Opus is the leader in Julia using the test specifications here. Test submissions are welcome!
My stack is a little clumsy right now because I haven’t quite dialed it in, but I use
Cursor as my editor. It’s a fork of VS Code with really good AI tooling. I do a lot of front-end stuff as well and it’s really nice to go to a a new file, hit Ctrl + K, and ask for exactly the file I want. You can even provide links to docs, other files, etc. to use as context. Very good tool, I shelled out for the pro version. You get Claude Opus calls for code gen and chat.
Claude Pro for very long architectural discussions or lots of guided code discussions, mostly because I use a lot of calls and I like the UI. I also do lots of non-programming chats with Claude, though using Claude via Cursor is better. I just haven’t nailed down the workflow yet.
PromptingTools.jl for misc LLM calls, mostly using Ollama as a backend with OpenAI for paid calls. I would switch to Claude but I just haven’t set up the API key and I am lazy.
AIHelpMe.jl for Julia-specific stuff when I’m in the REPL. It’s good shit and we should use it more.
What’s yours? What are you noticing as pain points or as things you really like?
I use ChatGPT Plus for lots and lots of things. It is really surprisingly good. I would consider Claude but do not expect the differences to be large enough to pay the “switching costs”.
I am trying out Cursor instead of VS Code. I have not yet found it to be entirely helpful and often jump back to ChatGPT to get AI help for something. It does offer nice autocomplete but it often misses the mark. Likely I just need to learn a bit more about all of its features and how to use them.
IMO the interface for ChatGPT is much better than Claude’s. The difference is not terribly large.
This is my sense too. I think I need to learn to be more competent with the chat functionality in the editor, but I find the fact that the chat history gets wiped almost every time to be a little discomfiting.
I’d love to run this as a full-on community survey for my JuliaCon talk!
But for the time being, here is my response:
Cursor - if you haven’t tried, you don’t know the power of auto-completion in the middle of the sentence! I can never go back to Copilot… And all the things Cameron mentioned!
ChatGPT Plus (just images for the in-image editing)
PromptingTools.jl with Groq Llama70b as the default (part of my startup.jl)
AIHelpMe.jl - only when I work with AlgebraOfGraphics/Makie, I get it running automatically, so it’s warmed up
ProToPortal.jl - personal mini-project that I actually love! I missed a GUI with MY templates in my stack and I love to use it when I cannot be at my computer (deployed on fly.io) + it’s much faster than ChatGPT and I’m an impatient person.
For a free alternative for people (not companies), Codeium works well.
In general, though, I’m inclined to using chatbots, rather than integrating the tool inside the code editor. I suspect that reading non-relevant suggestions are slowing down my work, rather than increasing my productivity. Still trying to find an approach that works perfectly for me, but not there yet.
I’m also curious to know what this looks like, almost a year later – whether more people are using AI coding assistance, and what the best usage patterns today look like. Especially because most chatter I hear online is about people using these tools for frontend dev, infra, etc – quite different from what I use Julia for, which is almost completely numerical programming, simulations, etc.
I’d appreciate hearing about experiences, or even just pointers to high-quality resources.
As for me: I was mostly only using VSCode + LSP previously. Only recently started testing the waters with Copilot and Aider. Might try out Cline/Cursor at some point.
@svilupp first off, thank you so much for the great work you’ve been doing with the LLM-related packages I would use PromptingTools.jl more often if it had a CHAT REPL mode where users could type a special character like / and start chatting without the ai"message" boilerplate:
/ # typing / as the first character in the line activates CHAT REPL mode
chat> What is the capital of Brazil?
🤖: Brasília is the capital
You can create custom REPL modes in Julia with packages like ReplMaker.jl. The example they give with a LISP REPL gives an idea of the end-user experience. It is really nice.
But they are a bit outdated now. IT would be great if someone wanted to develop quick & ergonomic REPL mode with PromptingTools!
Alternatively, it could be extended to use AIHelpMe.jl “RAG” which also leverages the available docstrings for your loaded packages or pre-computed knowledge packs to improve AI answers.
Please correct me if I am wrong, but isn’t AiHelpMe.jl equivalent to PromptingTools.jl + Julia documentation knowledge base?
If that is the case, I would focus on PromptingTools.jl and give it a single command PromptingTools.digest(url) to digest publicly-hosted knowledge bases. The digestion process could then use DataDeps.jl or similar package to store the knowledge base locally as an artifact. The REPL mode could then load these knowledge bases automatically, and users could configure their knowledge bases using environment variables or some other mechanism.
PromptingTools.jl → tooling to build a RAG-based system
AIHelpMe.jl → stateful application for specific use case (manages processed knowledge - via artifacts, configuration for the use case, extra utils to allow users to process loaded packages, easily change config, etc.). there are other packages supporting the use case (DocsScraper.jl)
I think it would make it quite bloated to move all the application-specific stuff into PT.
I see REPL mode as a front end that anyone can add easily, in the same way how Pere created a GUI with Stipple: GitHub - BuiltWithGenie/PkgAIHelp
I would use PromptingTools.jl more often if it had a CHAT REPL mode where users could type a special character like