Teaching AI To Test
Hey folks, for the past couple of months I’ve been discussing how we as developers can enter the QA mindset. For this next series, I’m delving into the world of artificial intelligence to talk about how we can help agents be better testers.
Let’s start with an obvious questions, why do we even want to do this? After all, with a little prompting an agent is more than happy to write and run unit tests for every change it makes. Do we really need to invest in any more capabilities than that?
The unfortunate reality is that unit tests are insufficient to cover the vast majority of production-grade software. Unit tests are fast and stable, which is why they serve as the foundation of the test automation pyramid. However, they are inherently limited in scope. If you have a micro-services architecture with high unit test coverage and low integration test coverage, then you likely won’t encounter many bugs within a single service, but you will be plagued with outages as services shift underneath each other. As any QA engineer will tell you, then only way to be certain the entire app will function is to test it as a whole.
A lack of effective testing often leads to reduced productivity. What I personally have observed is the phenomenon where AI-assisted development results in code being written quicker, but the pull requests for that code receive more pushback and ultimately take longer to land. In order to maintain a high standard of quality and achieve the promise of higher productivity through AI-accelerated development, we have a few options:
Hire more QA engineers to test that code.
Shift our developer workforce to be more testing focused.
Have the agent writing the code perform better testing of its work.
Plenty of discussion could take place around all three of those options, but for the purposes of this series I’ll focus on the practical steps to improve how an agent tests its code.
Now that we have an understanding of why we would want to impart better testing skills to agents, let’s quickly talk about why we even need to be the ones providing those instructions. After all, an agent fundamentally understands how to build an application, so aren’t Anthropic and OpenAI giving these models everything they need? What I’ve found is that the hardest part of enabling AI to test its own changes is not telling it how to test, it’s telling it how to test my app. Claude fundamentally understands what a feature flag is, but it doesn’t know about my admin dashboard for managing feature flags in the current session. We as developers have built tools and learned tricks that make testing much more efficient. Preloading that knowledge into an agent is essential if we want it to move as fast as a human.
In the next few posts I’ll walk through practical steps for teaching an agent about the intricacies of testing a given application, tips for organizing those learnings to be AI-ready, and the pitfalls of leveraging a machine as a substitute for human testing.
I'm curious about how the coding agents do with multi-threaded programming and testing, and then, multi-process system testing. I've done a lot of both of these, and both the writing and the testing. In fact, I've authored (and taught a lot of) C++ classes on both and you see a lot doing that. Once, I had a colleague tell me his solution for a tricky multi-threading challenge _was_ _correct_ he said "because I tested it". I had to point out to him that you really cannot _prove_ correctness of multi-threaded code by testing -- you can only "prove" that you were unable to break it under controlled circumstances :) Literally, multi-threaded code will fail because of milli-second (ok, micro?) differences in conflicting data access by competing threads often (usually) under OS control, and NOT your own. In addition to thread-conflict causing logic errors, mutexed code can straight-up deadlock. Oops. And yet, the last thing you want is heavily, needlessly mutexed code due to its performance penalties. (Which is one reason, in Java, I shy away from the synchronized collections - they promise to be thread-safe, but I feel it's often possible to encapsulate such actions in one's own crafted "owner thread" and judiciously manage the mutexing. Mutexing is expensive, did I say?) So, referring back to my "teaching" days -- I always told my students they must master these concepts because "you cannot prove multi-threaded code to be error-free through testing -- the quality has to be designed in from the start." (That said, nothing wrong with driving the logic through a million iterations or so looking for trouble!) So, as I began, I wonder how the agents handle all this, especially standing up multi-process system testing.