Recently, I read a lot how great AI coding agents are. I was little bit skeptical, but I thought it’s high time to give it a shot.
Goal: develop something useful related to robotics without writing a single line of code myself. Normally, I wouldn’t totally restrain from writing the code myself, but I wanted to test first how agent can work under my supervision, not as a pair programmer.
To make it even harder, I decided to use Julia, which has limited code base available. I did it also because I had some project ideas in Julia already researched.
Effect is RigidBodyCollisions.jl package GitHub - ljuszkie/RigidBodyCollisions.jl: Experimental AI developed collision checking extension for RigidBodyDynamics.jl
It is based on RigidBodyDynamics.jl and DifferentiableCollisions.jl and provides a couple of things:
- Parsing URDF collision tags, creating data structure for collision bodies associated to links
- Mapping simple shapes to the corresponding DifferentiableCollisions.jl objects, replacing meshes with bounding capsules, fitted with PCA and optimization
- Updating collision objects poses according to MechanismState
- Detecting self and environment collisions
- Custom gradients seamlessly integrated with ForwardDiff.jl
- Visualization with MeshCat
- Example use case: Collision free optimization based inverse kinematics
First of all this was pretty radical experiment in a sense that I didn’t write or change any code myself, everything was done by agents. Only files edited by me were implementation_plan.md, CLAUDE.md and AGENTS.md. I gradually added point after point to the implementation plan and asked agent to plan implementation of a specific point and then execute the plan. Of course many times result was not good enough, so it required additional prompting, unfortunately not captured in the repo. I also many time reverted the failed changes and either modified or split tasks in the implementation_plan.md, you can see that some points have a, b, c,… subpoints.
Another important points is that it was a project that I had previously researched, so that I had an idea how to do it, what libraries should be used and so on, it was not “Claude implement robot collision checking in Julia”.
Here’s a short list what I observed and learned:
- Good work break down is crucial, small, specific, testable tasks have much higher chance of successful implementation
- Unit tests are a must, it’s good to also require some user facing usage examples or demos, that you can easily run yourself
- Following the above point be sure to not only review generated code, but even more importantly the unit tests. Agents like to hide that they failed by either doing “Position test fails, let’s increase tolerance to 1m” or just removing some tests. Maybe next time I should try the more orthodox TDD and start with generating tests as the first step.
- Claude is very overzealous (Codex significantly less), so it is very important to specify what it shouldn’t do at this point to prevent it from jumping forward.
- Use plan function, it’s a good way to prevent it from going into wrong direction.
- If it struggles with implementation it’s usually better to check what might be the problem, revert and start again with enhanced prompt, for example adding “use high level API from xyz.jl”.
- Agents sometimes don’t follow instructions from CLAUDE.md or AGENTS.md, I think that’s a more common problem, I experience it sometimes with Perplexity spaces as well.
That would be it. Main conclusion is maybe not surprising, but important. You get way better results with AI assisted coding if you follow the best engineering practices, especially when it comes to breaking work into tasks that are small, well defined and with testable results. Who would think that, hah? 