Update: I’ve had good success using Documenter to iterate on Literate.jl scripts, to generate notebook-like analyses. Combined with julia-repl + Revise, I get fairly snappy updates to my reports.
The idea is to setup Documenter.jl normally, and tell the agent to write its Literate.jl scripts in /docs/src/notebooks. The agent can then call this function:
# For fast interactive iteration: include("docs/make.jl") once, then call
# preview_page("my_notebook") to rebuild just that page.
function preview_page(page::String)
outdir = joinpath(DOCS_DIR, "src", "generated")
mkpath(outdir)
Literate.markdown(joinpath(DOCS_DIR, "src", "notebooks", "$(page).jl"), outdir; documenter=true)
makedocs(
root = DOCS_DIR,
sitename = "Documentation",
pages = ["Home" => "index.md", titlecase(replace(page, '_' => ' ')) => "generated/$(page).md"],
)
end
to run that and generate the corresponding HTML, which it can then show in the browser. As a bonus point, those Literate.jl scripts are now ready to be included in the permanent documentation.
I use a little script to hijack julia call into a DamonMode.jl routine, so that neither do I need to teach AI to do anything, nor got paid for the JIT cost. One thing that is frustrating that I don’t usually get a good command of the daemon lifetime, but I haven’t found any problem yet.
Then how do you look at that notebook? Do you have a Jupyter server running permanently, and you open / reload that notebook once the agent has finished running it?
For me the next step is figuring out how to share the reports inside a pull request (as material supporting the change). If github supported rendering of static HTML, it would be much easier, but it doesn’t. With Literate.jl + Documenter rendering HTML, I think that I can setup LiveServer.jl (or maybe github pages?) to share the reports that way, but it feels quite round-about.
Well, I usually have it do all the iteration / exploration / debugging in a REPL, and then write up the notebooks with that exact code once it’s validated.
I’m very close to releasing KaimonSlate.jl which builds on top of Kaimon (though it’s not 100% necessary for the notebook interface). My goal was to create an environment conducive to AI/human interaction within the Julia ecosystem focused on cell based document creation and publishing.
If you’re interested in some beta testing, I could add you as a collaborator. The code is currently private in GH while I continue putting some polish in, but I hope to release this 'very soon™ '.
I’m currently waiting on the 3 day timer in order to get a slimmed down ‘gate’ package for Kaimon which will allow me to release Kaimon 2.0, which will then allow me to publish Slate.
If you would like to see some sample output, I’ve been working on the publishing code and I have some examples in a portfolio page here:
If anyone is interested in a sneak peek into Slate, I’ve made the repo public and docs are available:
Keep in mind that the docs are still in need of some completion and polish, and that I’m still working on features before the release. But if anyone would like do some beta testing (Windows users welcome since it’s not my testing environment), get in touch?
I’m trying to understand how KaimonSlate can interact with agents running outside of it, rather than just coordinating multiple agents inside a single notebook.
As context, I run a herdr-based workflow where agents coordinate by sending keystrokes to and read other panes. A test agent might evaluate how a model works as part of larger algorithm/pipeline via a harness, then hand off to a second agent to improve the model based on the results, finally waiting for a standard “I’m done” signal from a watcher before testing the new model. Most of the Julia evaluation is handled via Kaimon, messaging between panes is only used for coordination between agents.
Concretely, can KaimonSlate set a monitor or watcher that fires when an outside process signals completion, so a KaimonSlate agent could take part in that same handoff protocol with an agent running elsewhere? It feels like KaimonSlate could work very well as a supervisor agent of sorts that helped quickly crunch numbers and do quick visualizations, then delegating work to other agents outside of it.
I have a couple related unreleased projects which might scratch that itch. We should connect on DM perhaps to discuss.
But as for Slate specifically, I don’t have something built in that does this, but it would be easy to construct it. I don’t know your exact use case but I could take a stab what I think you’re getting at and show you a demo sometime today probably.
You might also be interested in the work I’m focused on before release which allows Slate to define DAG based workflows which can operate across machine/network boundaries. Currently targeted at single servers, but thinking about/planning cluster support too.
This aspect will likely be the most ‘experimental’ part of Slate at release time, but I think I’ve got it at a point where it can be useful so I’m planning on putting it in. Work here has been what’s delayed the release slightly but I’m really planning on it the next couple days.
This caught my attention, do you have anything you can send me about the DAG based workflows, I use a DAG in my project. You can point me to paperwork as well. Ill read it.
I’m guessing that what @cstjean is pointing towards is the cost that’s paid when you have an AI agent that only knows how to run Julia code as executable scripts, each time starting a new process and incurring extra overhead from needing to recompile the code with Julia’s JIT compilation.
As for token usage, it’s a separate matter, but this development loop with bash commands and parsing text script output is not optimized from a token usage perspective either. These are a couple of the issues I’ve worked on addressing with the AI tools I’ve created. Kaimon allows the agent to more efficiently interact with persistent, long running Julia sessions and it also optimizes the input/output between the model and Julia in order to use fewer tokens.
Slate is the platform I’m currently working on which builds upon those foundations and offers a modern notebook development environment that completely accessible to the AI agent. A catalog of MCP tools are available for the agent to interact with your Slate notebook environment and collaborate with you on what you want to build.
If it sounds interesting, please feel free to reach out for more info or support trying it out. I’m close to officially announcing/releasing a 1.0 version, but it’s already a very capable platform, IMO.
Comic back to the original question: Normally, Kaimon works quite well for me. One issue I have is testing. What I would prefer:
to see the progress of the tests
Kaimon should use the already open session and not start a new session for the tests
execute the script with the new/ latest tests using ex and include
Background: At the beginning of the test scripts, I always activate the test environment.
Not using the already open session is very time-consuming: Precompiling my package might already take 90s. Not needed if the already open session is used for the tests.
Would it be possible to configure Kaimon to use this approach?
My goal with test run was to attempt to clean up and distill the IMO rather messy outputs of Julia’s default test frameworks, in order to present only the most salient information to agents (a token efficiency strategy). I feel like it does a decent job of this, but I’m sure it could still be improved.
This was actually originally the way the tool worked. I ended up switching to the current model because I found over time that running tests in the same warm Julia session had the possibility of behaving differently due to the existing state, or potentially corrupting that state or in some cases crashing and terminating the process. So I changed it to the current model where tests run in an isolated and fresh setting.
I could see introducing a configuration option which would let you choose whether to isolate or to run within the connected session. Would that help your use cases?
