AI error messages, and solutions/code fixes

Internals & Design
1

@tecosaur I was starting to read:

With many compilers you get a location like program.x:43:22 that you have to decipher. Where is that file? Which one is the line? Which is the column? Okay, let me scan through my code.

I’m all for improving error messages as much as possible. But it got me thinking, the compiler/Julia has all this information. It’s only showing line, because showing the full function is too verbose. Or all up the call chain…

But that could be very helpful for AI, so can I nerdsnipe someone to make a package, that calls an AI with all the needed info, and ask the AI to suggest a fix, and show short output from AI at the end of the error message/stacktrace (from its much larger input with context)?

I post the suggestion here under Internals, not because I want it in Julia right way. It can be prototyped in a package first (and we’ve seen problems with adding stdlibs prematurely, and Kristoffer wanting to revert one).

But maybe eventually the package could be integrated into Julia and/or VS Code extension?

FYI started reading at:
@seelengrab (can’t link your URL with spaces): Error Shrinking: The Good, the Bad and the Ugly

My understanding is that Julia error messages have already been changed to help AI, with short addition for it, as a hint, I’m just thinking way more help for it (for when it doesn’t actually have all the context already, as in it made the code), for calling an AI API (or for local AI model). The infrastructure would even be used for working code, e.g. when it’s type unstable, the AI could even bet @code_warntype and @code_lowered or even @code_native or whatever might help (output from, external tools, JET.jl, Aqua.jl?), even let it give help with optimization, and detecting global vars used accidentally.

[Has anyone seen such implemented; e.g. for other languages? I’ve already had some success with just showing AI error messages; often with the context since it made up the code (yes, at first broken) anyway in Julia. This is something people have also claimed for other languages, in fact the AI, does this with agents I believe, not yet tried such with Julia or other languages. Quick question, any suggestion for that, MCP is the thing? Good for Julia too?]

I find related thread:

2

Even putting aside capability and hallucinations, integration would be a terrible idea. Base is bloated as-is, and incorporating APIs for often-proprietary software would make it a lot worse. It’s a bad interface to AI because users tend to be very opinionated about their choices, and maintaining enough AI backends within Base or VS Code directly would be a nightmare. None of that is necessary when the prompt only needs the text of the call, stacktrace, the codebase, and maybe some static dispatch analysis like JET/Cthulhu. This should be third-party, likely model-specific packages with some shared dependencies for retrieving information for the prompt; Base already provides err for the last thrown errors, and catch_backtrace can be used in catch blocks.

3 Likes
3

I know that some programming languages have integrated AI-friendly error messages into their design, such as moonbit language, which is better suited for use with AI. It comes with a built-in AI assistant.

4

I suspect that there’s enough overlap between “useful for AI” and “useful to people” that we can improve both by just focusing on the latter, given the status quo.

2 Likes