We Audited Our Own AI Coding Agent

It is easy to demo an AI coding agent. It is hard to trust one with a live token to your code, a shell in a cloud machine, and the ability to open pull requests. The gap between "works in the demo" and "safe to hand a repository" is full of unglamorous edge cases.

So we ran a structured audit of CloudAxis Coding across five areas — billing, model selection, environment handling, delivery, and security — and fixed 22 issues before calling it a public beta. This is an honest tour of the interesting ones, because the willingness to publish them is part of what "trustworthy" means.

Why audit before launch, and publish it

Most software is marketed on what it does when it works. Infrastructure that holds your credentials and touches your code has to be judged on what it does when things go wrong — a network blip mid-push, a malicious README, a session that crashes halfway. Those paths do not show up in a demo. They show up in production, on someone else's repo.

We would rather find them ourselves and tell you than have you find them. Here are the ones worth sharing.

Billing that charges for exactly what ran

The most consequential bug was invisible from the outside: coding sessions were being priced as if they ran on a cheaper model than the one that actually executed them — a systematic under-charge. That is a bug in the customer's favor, which is precisely why it is dangerous: it hides, and it would have distorted every cost number we and you rely on.

We fixed it so the model a session is billed for is always the exact model it runs on, and hardened the accounting so per-step costs are recorded to the fraction of a credit instead of rounding sub-credit steps to zero. Trust in an autonomous tool includes trusting the meter.

A failure should never be silent

We found that a session which failed — say a push that hit a network error — could go quiet: the work existed in the cloud machine, but you were never told, and it could be reaped before you noticed. For an async tool you are not watching, silence is the worst possible failure mode.

Now every failure notifies you on your channel, a push that hiccups is retried before giving up, and the message tells you where your work is. Along the way we also caught a latent bug where the "reply YES to merge" code was being generated but never actually delivered — so the one thing you needed to ship was missing from the message. Fixed, with a test that keeps it fixed.

The auto-fix loop that did not loop

CloudAxis can pick a session back up when CI fails and push a fix. Except we found it was re-attaching to a cloud machine that had been stopped between runs — and then failing against a dead sandbox. The feature looked implemented and passed its tests, because the tests used a fake machine that did not model being stopped. Classic green-tests, broken-in-production.

We fixed the resume path to restart the machine first, added the missing test, and while we were in there, plugged a related leak where a machine could be orphaned if a session crashed mid-setup. Cloud compute that quietly keeps running is both a cost and a trust problem.

Treating your repository as untrusted input

The agent reads your repository's contents and runs shell output — and a malicious file or dependency can try to steer a model. We do not pretend a heuristic can "stop" prompt injection; the real controls are structural. But we added an audit trail so an injection attempt in repo content is logged the same way a suspicious GitHub comment is, and we made the highest-risk requests — trying to exfiltrate secrets or bypass the review gate — get refused at the door instead of started and cleaned up afterward.

We also closed a denial-of-service hole where a small, highly-compressible uploaded archive (a "zip bomb") could have exhausted memory before size limits kicked in, and scrubbed any credential from the machine's git config so the agent cannot read a token back out of it. Boring, important, done.

The proof gate, and honest test badges

Two fixes were about not lying to you. The merge gate that is supposed to refuse "unproven work" was only checking that a proof step ran, not that it captured anything — so an empty proof could slip through for a web app. And the "tests passing" badge could be painted by a command that merely mentioned a test tool. Both now require the real thing: genuine proof when a preview was possible, and an actual test invocation before any green badge.

An AI tool that overstates its own confidence is worse than one that admits uncertainty. We would rather show you "tests were not run" than a green check that means nothing.

What this means for you

Public beta does not mean finished — it means good enough to trust with real work, honest about the edges, and improving in the open. We fixed 22 issues, wrote tests so they stay fixed, and had a second review pass catch two more that we fixed too (including one that had silently turned three of our audit features into no-ops).

If you are going to give an AI agent a key to your code, you deserve to know it was pressure-tested first. Now you do. The rest is on us to keep earning.

Frequently asked questions

Is CloudAxis Coding secure enough to give a repo token?

It uses short-lived, repo-scoped GitHub tokens in isolated single-use machines, screens untrusted input, refuses high-risk requests, and requires human approval to merge. Like any beta it is improving; we publish what we find and fix.

Can I read the full audit?

This post covers the highlights. The point of publishing it is that a tool trusted with your code should be transparent about how it was hardened.