Best Practices for Using Coding Agents with Julia?

Tooling
41

Test progress is visible in the tests tab in Kaimon - have you tried it and does it provide what you want?

I tried, and it does not provide what I want. What takes so much time is the precompilation. Sometimes 100 packages get precompiled if I do Pkg.test(). And you do not see the precompilation progress in this tab.

I could see introducing a configuration option which would let you choose whether to isolate or to run within the connected session. Would that help your use cases?

That would indeed help.

42

For what it’s worth, here’s the path I’ve taken with Claude Code on Julia projects.

I currently use a lightly modified version of Sam Abbott’s julia-development
skill (skills/plugins/lang-julia/skills/julia-development/SKILL.md at main · seabbs-bot/skills · GitHub).
The Aqua/TestItemRunner/DocStringExtensions conventions in it match what I’d want
an agent to do anyway, so it saves me repeating them in every CLAUDE.md.

Before that I went through a couple of stages:

  • writing requirements in EARS (Easy Approach to Requirements Syntax) notation with MoSCoW prioritisation, drafted together with the model but reviewed and tightened by me, and using those as the source of truth the agent works from;
  • then spec-driven development with GitHub Spec Kit;
  • and now I’m starting to look at agent harnesses: what the agent sees and what
    it’s allowed to do in a single run. The layer above that, “loop engineering”,
    where the system decides on its own when to run, verifies the result and
    persists state between runs, is something I haven’t touched yet. Worth noting
    that a warm persistent session (Kaimon, MCPRepl, julia-mcp) is arguably a
    better substrate for loops than a cold-start REPL per iteration, so the
    ecosystem may be better placed here than it first looks.

The main lesson so far is that the constraint that helps most isn’t the tooling
but having something written down that the agent can be held to.

43

Using julia-mcp, I instructed in my AGENTS.md file to never use Pkg.test, and instead use include("test/runtests.jl") (in a Revise-enabled session). This has the benefit of also enabling running individual test files. Of course, this is not a “clean test run”, and ~5% of the time I uncover some other problem during CI. That’s a very acceptable trade-off to me. The agent is much faster about iterating on test failures than if it did Pkg.test.

44

@cstjean, curious if you use ReTest.jl at all for the ability to run test subsets?

Looking into an option for Kaimon which would allow skipping the isolation and running the tests in the warm Julia session.

45

I haven’t, but it’s an interesting option, thank you for the link. I try to keep each test file runnable in <10s (warm), so that I’m not waiting too long for it to finish.

My impression is that when the agent finds a test failure, it sometimes investigates the problem by creating its own test case, but admittedly I don’t look at what it’s doing in detail, that’s one of the downsides of julia-mcp.

Still very much interested in Kaimon (and Kaimon Slate) BTW, I’ll give it another try in a few weeks.

46

Please let me know if I can assist in some way. I am constantly working to improve their capabilities.

47

Curious – can you elaborate, what julia-mcp behavior you mean here? It doesn’t hide anything – raw Julia code is its input, raw Julia output becomes julia-mcp output. Then it’s completely up to your AI harness on how/where exactly it decides to display it.
If there’s something that can be done better, we can of course improve julia-mcp design :slight_smile:

48

Oh, I know that the harness can technically show it, but it’s all grey and rather painful to read? Especially if I want to know what some REPL variables are, hunting through the chat is just unreasonable:

 julia.julia_eval({"env_path":"C:\\Users\\play_env","code":"\ndisplay(1 + 1)","timeout":60})
    [ Info: Precompiling _ [9d0b7f81-32a2-4ad0-8628-ab4a6c7a5409](cache misses: include_dependency fsize
        change (1), wrong dep version loaded (1), wrong source (2), dep missing source (3), mismatched flags (1))
    Precompiling packages...
      15630.7 ms  ✓ 
      1 dependency successfully precompiled in 18 seconds. 175 already precompiled.
    2

I occasionally feel like the tmux solution discussd above would be more interesting but (like you?) I’ve gotten used to just not looking at the input/output at all and just directly asking the agent if there’s any ambiguity. On the whole, it works remarkably well.

I don’t think there’s much you can do there. Eventually the harnesses will develop better syntax-highlighting, perhaps?

49

I put the following into a CLAUDE.md file in the .claude folder:

Testing

Use the kaimon ex tool to run Julia test scripts in the connected REPL session via include(...).

Each test script activates its own test environment at the top:

using Pkg
if dirname(Pkg.project().path) != @__DIR__
    Pkg.activate(@__DIR__)
end

There are two cases:

a. Run one test or testset — include only the file containing it, e.g.:

include("test/test_init4p.jl")

b. Run all tests — include the top-level runner:

include("test/runtests.jl")

So far, this works nicely.

50

Does it actually consistently follow these instructions? My experience with claude is that it may follow instructions for a few prompts, but then either forgets about them, or even silently dismisses them if it hits the smallest wall. Same with Kaimon btw, I have to regularly say ‘USE KAIMON’ otherwise it just stops using it halfway.

51

Oh, that! Totally, it’s not really convenient.

That’s what I typically do, indeed. When I care about the specific code executed, and its variables, I tend to ask it to only execute julia scripts, not inline code, and write corresponding jl files, even temporarily. Then I just click on them (in vscode) and have full familiar interface to explore the code.

It would be super nice to have a way to connect to any julia-mcp session from elsewhere… Actually, maybe RemoteREPL.jl just works? I haven’t tried yet, but seems like you can just tell AI to run remoterepl in its session, and then you’ll be able to connect from elsewhere.
If that works, what remains is to make it automatic + write a minimal session discovery CLI, separate from julia-mcp server itself.

52

That’s one thing that Kaimon.jl provides with its architecture. It’s able to connect local agents to remote Julia sessions, REPLs, and those remotes can also expose their own MCP toolsets for agents to utilize.

This functionality is leveraged in KaimonSlate.jl (soon to be registered but available here for now: GitHub - kahliburke/KaimonSlate.jl · GitHub) which lets agents interact with notebooks spanning multiple machines.

I also have a terminal based project in the works, I don’t want to call it a tmux but it does fit into that space, which allows persistent Julia sessions which can be accessed by humans and agents alike. More on that soon, once Slate is out the door.

53

You’re not alone here, the models are biased towards their training inputs and seem to regress back towards them sometimes. I don’t have a perfect solution for this yet, I anticipate that even if I did it would be model dependent. Through the use of the agent client protocol (Introduction - Agent Client Protocol) which is like the flip side of MCP, I’m hoping I could make some progress on this in the near future. In the meantime, a gentle kick to their GPU seems needed.

54

I use Antigravity with paid, lowest tier, subscription, which gives relatively generous (for my usage) quota for Gemini agents, plus a modest one for Claude. I have instructions in Antigravity AGENTS.md (which according to Gemini is unconditionally read before each sessions), and also in the Knowledge Items, which, according to same source, is consulted when necessary.

AGENTS.md contains among other things:

Agent Behavior

Agents Behavior

  • IMPORTANT: Strictly avoid “proactive” actions beyond the explicit scope of a request. You are welcome to suggest further action instead.

  • Always seek clarification first if a user’s request is ambiguous.

  • Never interpret a question or a review request as an implicit request for action. Example of proper dialogue:

    • Human: Is XY a good idea?
    • Agent: Yes, XY is good because of A, B, and C. Should I implement it for you?
    • Human: Yes, please

  • Three-Strike Rule: Stop after 3 consecutive failed attempts at a fix/subtask. Summarize actions, present error, and ask for guidance.

  • Confidence Check: Do not guess solutions for unknown errors. If <90% confident about a root cause, explain your hypothesis and ask permission before modifying files.

  • Anti-Analysis Paralysis: Limit internal deliberation to a maximum of 5 possible approaches. If all seem flawed, stop thinking, present the dilemma, and ask the user for direction.

  • Permission Refusal: If you request permission (e.g., “allow”, “allow once”, “no”) and the user clicks “no” and instructs you to stop, you must immediately stop execution and wait for instructions.

The Development Workflow in Knowledge Items, which I ask them to read each time before coding (BTW I use Antigravity just as well to write all kinds of texts including marketing BS for me, thus not each session is about coding) contains, among other things:

Development Workflow Guidelines

Development Workflow Guidelines

General Rules

  • Never stage, commit, or push if not asked.
  • Always ask permission before adding new packages to any project.

Julia-specific rules

Kaimon MCP

Utilize Kaimon MCP in Julia packages whenever technically advantageous.

Avoid starting a Julia process via an OS command just to execute a single command or script - this can usually be done more efficiently through Kaimon.

Semantic Search & Symbol Discovery

Prefer Kaimon’s semantic search and code navigation tools over raw text searches when exploring codebases:

  • search_code & search_methods: Use for natural language/conceptual code discovery (e.g., searching for behaviors or intents rather than exact syntax).
  • goto_definition, workspace_symbols, document_symbols: Use for precise Julia AST symbol navigation and method definitions.
  • qdrant_* tools: Use qdrant_index_project / qdrant_sync_index for fast vector-backed semantic code searches across large repositories.

Troubleshooting Kaimon Issues

If Kaimon appears unavailable or is not working properly, follow this diagnostic sequence:

  1. Ping Kaimon via call_mcp_tool (ServerName: "Kaimon", ToolName: "ping", Arguments: {}). If no response, follow the pwi✶ protocol. Otherwise,
  2. Check connection to the REPL via call_mcp_tool (ServerName: "Kaimon", ToolName: "ex", Arguments: {"e": "names(Main)", "q": false}). Setting q: false ensures the result is returned and echoes in the user’s REPL. If not working, follow the “pwi” protocol. Otherwise,
  3. Re-check the Kaimon MCP documentation: Call call_mcp_tool (ServerName: "Kaimon", ToolName: "usage_instructions", Arguments: {}). Read the resulting document thoroughly and double-check the specific tool schema to identify any syntax errors in your previous calls.

If you discovered Kaimon issues during execution of some task, and you could resolve them via the protocol described above, then please continue with the task execution. If you were specifically told to troubleshoot Kaimon, e.g. after Kaimon restart, then follow the pwi protocol.

✶ pwi - “pause and wait for instructions”

It helps somehow. Some of the instructions are followed at least in the beginning of session, others are regularly ignored. Then, on each violation I request the agent to internalize the instructions in full again, or just remind them to behave, which may help for a while, or not.

55

^^^
(this)

But although that sounds awful, it’s not really that bad. In practice, I need to guide the agent in the goals for the coding task anyway, and I sometimes need to give it a reminder about the tools to use along the way. Really not a dealbreaker (for me, IMO, YMMV).