Teaching AI To Test: The Cost

Hey folks, today I’m wrapping up my series about teaching AI agents to test your application by examining the cost of this process from a few angles. This blog post will drive toward the fundamental question, should you even be spending resources on a project like this?

In my last post, I quoted a few token counts for testing a relatively straightforward ticket. While implementing the feature from my example cost ~20k tokens, testing it cost between ~60k and ~97k. How much is that in real money? Difficult to say. A token-based cost structure intentionally obfuscates dollar amounts. Furthermore, the agents themselves are typically isolated from the expense side of their runtimes. The next time you have Claude Code complete a task for you, ask it how many tokens it just used. When I tried this, my agent had no idea, and told me it wasn’t capable of accessing that sort of information. The Claude Code runtime will track usage and write it to the terminal which is useful… if the session isn’t interrupted in any way… and if you’re running this on a persistent system, not an ephemeral environment. In order to get my token counts I had to run locally on my laptop and monitor active usage myself.

So we don’t know exactly how much it costs to test a ticket, but we can say that in my experiment it cost roughly three times as much as actually implementing the ticket, even when making an active effort to improve efficiency and decrease token spend. That’s a starting point, but still not the full picture. In my previous post I also alluded to the fact that this was acceptance testing. A subagent read the acceptance criteria from the ticket and ensured that the changes met that criteria. However, as we’ve discussed before, acceptance testing is not a high leverage use of novel thinking in testing. How much more would it cost to have the agent conduct exploratory testing? How much additional context would it need to do so effectively?

That last question gets to the heart of the matter. I said from day one in this series that the difficult part of this work is not teaching an agent to test an application, it’s teaching an agent to test your application. Not only that, you have to teach every agent to test your application. Each agent comes in fresh, with a vast corpus of knowledge about how software works, and zero context about what you are building. It needs to read the manual left by a previous agent, review the work that’s been done, and attempt to complete the goal you set for it. If its context window is exhausted, it leaves a note for the next agent and poofs out of existence.

I’m reminded of the episode “Meeseeks and Destroy” from season 1 of the TV show Rick And Morty. It features creatures called Mr. Meeseeks that can be spawned into existence ad infinitum to assist their summoner. They have limitless knowledge and are eager to help complete simple requests, but they spiral spectacularly when they are applied to unstructured problem spaces. More than anything, they want to complete their task and stop existing. That last point is all too relevant when you read that 50% of agent roles in Gastown simply ensure other agents are doing work instead of turning off.

Why am I belaboring this point so much in a post about agent testing cost? Because it represents a deep gap in what we can expect from agentic testing versus human testing. It’s no accident that Quality Assurance Engineers often become lore keepers for their organization. Their constant exposure to the application, coupled with a much larger working context than any off-the-shelf agent has to date, means they know more about an application than most other people at a company. This experience matters. As we know from ISTQB’s seven foundational testing principles, exhaustive testing is impossible, testing is context dependent, and defects cluster together. Effective testing is always a budgeting act that requires devoting testing effort to the places that matter most. The most important input for deciding what to test in the future is understanding what failed in the past. It’s hard to have that context when you weren’t the one that experienced that failure.

Back to the question I posed at the beginning of this post, should you even be spending resources on a project like this? Ultimately, that is for every organization to decide for itself. My experience so far tells me AI testing is a viable option. However, going that route has both concrete and intangible costs associated with it. There’s no right way to do testing, only decisions and consequences. Hopefully this series has given you the insights necessary to make that decision for yourself.