Teaching AI To Test: The Organization
Hey folks, continuing from my previous post about teaching an AI agent to test your app, today I’ll outline ways I’ve organized those teachings for future agents. This post will largely focus on my personal efforts to create an effective and efficient testing agent, but general lessons can be extracted for your own work.
The naive organization, which is absolutely what I did on my first pass, is to heap all of the instructions you’ve generated into markdown files in the .ai directory of your repository. When you launch your agent it will automatically parse these instructions and add them to its context window. There’s a few problems with this approach:
That eats up context tokens for every agent you launch, regardless of whether you intend to use it for testing.
It leaves the decision of when to use those instructions up to the agent rather than providing it with clear guidelines and reminders to use them.
It breaks the most fundamental rule of entering the QA mindset by having the author agent test its own code.
During my first attempt, I asked an agent to implement and test a relatively simple feature change. It needed to update text in a toast notification of a website and place that new text behind a feature flag using our A/B testing system. Reading the Jira ticket, accessing the designs, and implementing change used about 20k tokens in Claude Code. The remaining work of testing the change and providing a recording as proof of that verification required another ~97k tokens, and the artifacts were pretty bad. What should have been a couple simple videos of displaying the toast in each experiment variant turned into a six minute odyssey of the agent roaming about the site intermixed with unexplained lengths of inactivity. What’s worse, when the agent opened a draft pull request of the change for review, a reviewer agent quickly identified an edge case that had not been accounted for.
Thus I began a complete overhaul of the structure of the instructions. Since I was using Claude Code I decided to create a skill as the primary entry point rather than writing a wordy prompt to kick things off. The skill was called /complete-ticket and it accepted a Jira ticket as the argument. This skill instructed the agent to complete the following steps:
Review the Jira ticket, associated designs, and relevant areas of the code.
Make the code changes necessary to complete the ticket.
Spin up a subagent that should complete a /user-acceptance-testing skill.
Address any feedback from the subagent, repeating step 3 if necessary.
Open a pull request.
Notice that I began leveraging a subagent for testing in step 3 instead of the author agent. This increased the distance from the code in order to address the problem of broken edge cases going unnoticed. The /user-acceptance-testing skill itself was another entry point. It began by instructing the testing subagent to review Jira for itself to determine if the code change met the definition of done for the ticket. It then told the testing subagent to create a test plan in a markdown file that it could refer back to. The plan was to include specific testing skills for each step. These concrete skills related to features of the application, such as /ab-feature-testing and /metered-feature-testing. By reorganizing the instructions as skills and telling the testing agent to create a disciplined test plan, I eliminated the wasted tokens on start up and ensured the testing practices were top of mind when the agent needed them.
With the new organization in place, I asked a new agent to pull the Jira ticket using the /complete-ticket skill. Once again the implementation used about 20k tokens. However, the testing portion burned only ~60k tokens this time. The verification evidence it produced were two 15 second videos showing the toast text in each A/B variant. When the pull request was opened, the reviewer agent found no missing edge cases in the implementation. Given the goal was to create proof-of-testing artifacts to boost developer confidence for an AI-generated pull request, the simple act of reorganizing instructions to be more AI-ready improved those outcomes by leaps and bounds.
Ultimately, this was a pretty exciting result to see, though I would be remiss were I to gloss over the cost of this process. The act of testing a change and iterating on the solution was three times that of simply reading a ticket and implementing the code. In my next post I’ll spend some time reflecting on this expense.
I find these notes to be some fascinating and useful findings for test. My domain and process are a bit different but there is much overlap. For example, missing corner cases or other required logic. Once, I had an Agent enhance code for me to introduce two-factor analysis with a second, parallel data source. The analysis code was written, but the input statements to access the second data source were not. (!) ... please try again ...
One other thing that I find useful, periodically, is passing the output from one agent as input to another (completely different) ... and see what happens next. I've had a number of varying responses using this technique from - "This code will never compile" [yet it did and when given the output, the response shifts to - "Well, I see, but now I see the results are semantically incorrect for these reasons..."] ... and on to ... "Well your data model [remember, defined by the prior Agent] results in 'severe overfitting' - so changes are imperative."
Certainly, every Agent response cannot be cross-checked (endlessly?) in search of such dissenting (and disparaging :) reactions. But I feel that it pays to check, from time to time, for "a second opinion."