The naive organization, which is absolutely what I did on my first pass, is to heap all of the instructions you’ve generated into markdown files in the .ai directory of your repository. When you launch your agent it will automatically parse these instructions and add them to its context window. There’s a few problems with this approach:

1. That eats up context tokens for every agent you launch, regardless of whether you intend to use it for testing.

2. It leaves the decision of when to use those instructions up to the agent rather than providing it with clear guidelines and reminders to use them.

3. It breaks the most fundamental rule of entering the QA mindset by having the author agent test its own code.

During my first attempt, I asked an agent to implement and test a relatively simple feature change. It needed to update text in a toast notification of a website and place that new text behind a feature flag using our A/B testing system. Reading the Jira ticket, accessing the designs, and implementing change used about 20k tokens in Claude Code. The remaining work of testing the change and providing a recording as proof of that verification required another ~97k tokens, and the artifacts were pretty bad. What should have been a couple simple videos of displaying the toast in each experiment variant turned into a six minute odyssey of the agent roaming about the site intermixed with unexplained lengths of inactivity. What’s worse, when the agent opened a draft pull request of the change for review, a reviewer agent quickly identified an edge case that had not been accounted for.

Thus I began a complete overhaul of the structure of the instructions. Since I was using Claude Code I decided to create a skill as the primary entry point rather than writing a wordy prompt to kick things off. The skill was called /complete-ticket and it accepted a Jira ticket as the argument. This skill instructed the agent to complete the following steps:

1. Review the Jira ticket, associated designs, and relevant areas of the code.

2. Make the code changes necessary to complete the ticket.

3. Spin up a subagent that should complete a /user-acceptance-testing skill.

4. Address any feedback from the subagent, repeating step 3 if necessary.

5. Open a pull request.

Notice that I began leveraging a subagent for testing in step 3 instead of the author agent. This increased the distance from the code in order to address the problem of broken edge cases going unnoticed. The /user-acceptance-testing skill itself was another entry point. It began by instructing the testing subagent to review Jira for itself to determine if the code change met the definition of done for the ticket. It then told the testing subagent to create a test plan in a markdown file that it could refer back to. The plan was to include specific testing skills for each step. These concrete skills related to features of the application, such as /ab-feature-testing and /metered-feature-testing. By reorganizing the instructions as skills and telling the testing agent to create a disciplined test plan, I eliminated the wasted tokens on start up and ensured the testing practices were top of mind when the agent needed them.

With the new organization in place, I asked a new agent to pull the Jira ticket using the /complete-ticket skill. Once again the implementation used about 20k tokens. However, the testing portion burned only ~60k tokens this time. The verification evidence it produced were two 15 second videos showing the toast text in each A/B variant. When the pull request was opened, the reviewer agent found no missing edge cases in the implementation. Given the goal was to create proof-of-testing artifacts to boost developer confidence for an AI-generated pull request, the simple act of reorganizing instructions to be more AI-ready improved those outcomes by leaps and bounds.

Ultimately, this was a pretty exciting result to see, though I would be remiss were I to gloss over the cost of this process. The act of testing a change and iterating on the solution was three times that of simply reading a ticket and implementing the code. In my next post I’ll spend some time reflecting on this expense.

First of all, let’s talk about the technical setup. In the realm of web development the highest order testing is loading the application in an actual browser. You can quickly bootstrap a testing harness for this using Claude Code, Playwright, and the playwright-cli binary / Claude skill. If you also install Xvfb then this whole setup can run headlessly on a virtual environment, allowing you to manage work on multiple features at once.

Now that we have the tech sorted out we should talk about what we’re trying to accomplish. Let’s say our goal is to enable AI to work on features fully autonomously. With a single command we want an agent to complete the following steps:

1. Review the work ticket, associated designs, and relevant areas of the code.

2. Make the code changes necessary to complete the ticket.

3. Execute user acceptance testing.

4. Make any additional changes based on the testing outcome.

5. Record itself running a final verification of the code changes.

6. Open a pull request for human review.

7. Post the verification records to the pull request.

Steps 3 and 4 are intended to improve the quality of the change set prior to human review to limit the amount of time we sink into correcting the agent. Additionally, steps 5 and 7 ensure the human receives proof in the pull request that satisfactory testing was completed. For our web app example this takes the form of screenshots / screencasts from the latest testing attempt that demonstrate the feature meets the acceptance criteria described in the ticket.

So how do we give the agent the context necessary to perform that testing? You might be inclined to open a text editor and write page after page of instructions about how your feature works and the best ways to test it. Resist that urge. Instead, boot up Claude Code, ask it to launch Playwright, and instruct it to navigate to your application. Give it a guided exploration of your area of the codebase. For instance, try sending it prompts like these:

What do you think the purpose of this page is?

Notice there’s a button with a gear icon in the corner. What does that do?

Try switching to a different mode. How does that change the experience?

The following is the URL for our admin panel. Notice that is has the capability to reset metered features. How could that be helpful for testing?

Once the agent has a sense of how the app works at a high level, we can introduce it to some of the deeper inner workings like the database. After all, being able to insert a test account into a database is much faster and less error-prone than registering one through the UI. Connect Claude Code to your codebase and a development database—not your Production database—and explore the schemas and ORMs that underly your test data:

Inspect the contents of the Users table. Notice there are different types of users. How would that come in to play when testing a feature?

The following is the ORM definition for a User. How could you use that to insert a new user into the database?

Now that you’ve spent some time exploring your feature, ask the agent that did the exploration to write the skill for you:

We have a lot of useful knowledge about this feature. Write a new skill that captures all of that knowledge so future agents can use it as well.

Working through this process will give the agent a great start for testing your feature, but remember that setting up high-level test harnesses requires time and effort. Teaching an agent to test your application is no different. Many developers find Selenium-style tests hard to write compared to unit tests because Selenium-style tests are running on an environment that behaves in a non-deterministic way. Autonomous agent testing compounds the problem as the agent itself behaves in a non-deterministic way:

• Unit Tests: Deterministic tool, deterministic environment

• Selenium: Deterministic tool, non-deterministic environment

• Autonomous Agents: Non-deterministic tool, non-deterministic environment

This means the best advice for writing solid Selenium tests also applies to writing solid autonomous agent testing skills. You must run them over and over again to see what breaks. Concretely, follow these steps:

1. Identify a ticket that would require your new skill to test.

2. Have an agent write the code for that ticket, but have it pause at the testing stage.

3. Commit the code change from Step 2 to a branch.

4. Spin up a brand new development environment and launch a new Claude Code agent to ensure you’re starting with a clean context.

5. Check out the branch from Step 3.

6. Tell your agent it is on a branch with untested code changes for a particular ticket. Ask it to verify the change.

7. Observe the agent’s behavior, particularly what goes wrong.

   • Does it get confused when it sees a 500 responses from ancillary services?

   • Does it get stuck in a loop because it missed an error message?

   • When something goes wrong, does the agent have a useful fallback strategy to complete its testing?

8. Repeat Steps 4 through 7. Ideally run through this process five to ten times.

The more you battle test your skills against actual agent-generated code in actual development environments, the more self-sufficient future agents will be in their testing process.

With that, we now have a process to teach an agent how to test your application. In the next post I’ll dig into practical tips for organizing these instructions so they are understandable and actionable by agents.

Let’s start with an obvious questions, why do we even want to do this? After all, with a little prompting an agent is more than happy to write and run unit tests for every change it makes. Do we really need to invest in any more capabilities than that?

The unfortunate reality is that unit tests are insufficient to cover the vast majority of production-grade software. Unit tests are fast and stable, which is why they serve as the foundation of the test automation pyramid. However, they are inherently limited in scope. If you have a micro-services architecture with high unit test coverage and low integration test coverage, then you likely won’t encounter many bugs within a single service, but you will be plagued with outages as services shift underneath each other. As any QA engineer will tell you, then only way to be certain the entire app will function is to test it as a whole.

A lack of effective testing often leads to reduced productivity. What I personally have observed is the phenomenon where AI-assisted development results in code being written quicker, but the pull requests for that code receive more pushback and ultimately take longer to land. In order to maintain a high standard of quality and achieve the promise of higher productivity through AI-accelerated development, we have a few options:

1. Hire more QA engineers to test that code.

2. Shift our developer workforce to be more testing focused.

3. Have the agent writing the code perform better testing of its work.

Plenty of discussion could take place around all three of those options, but for the purposes of this series I’ll focus on the practical steps to improve how an agent tests its code.

Now that we have an understanding of why we would want to impart better testing skills to agents, let’s quickly talk about why we even need to be the ones providing those instructions. After all, an agent fundamentally understands how to build an application, so aren’t Anthropic and OpenAI giving these models everything they need? What I’ve found is that the hardest part of enabling AI to test its own changes is not telling it how to test, it’s telling it how to test my app. Claude fundamentally understands what a feature flag is, but it doesn’t know about my admin dashboard for managing feature flags in the current session. We as developers have built tools and learned tricks that make testing much more efficient. Preloading that knowledge into an agent is essential if we want it to move as fast as a human.

In the next few posts I’ll walk through practical steps for teaching an agent about the intricacies of testing a given application, tips for organizing those learnings to be AI-ready, and the pitfalls of leveraging a machine as a substitute for human testing.

The first example comes from a job many years ago. I worked at a two-sided market place, so our users were both people buying things and people selling things. The sellers were a particularly lucrative target for scammers. The ploy was simple: a scammer would create an account with an avatar and name that appeared to be a site admin, send the seller a message that a sale had failed to process, and ask the seller to provide updated bank account information. I was on a project that was trying to prevent those scammers from contacting our sellers.

The messages started off easy to identify. The body was plain text, the username was overt, the avatar was simply our company logo. We built an AI model that classified these as spam, and the rate of sellers opening these messages dropped precipitously. In response, the scam messages steadily became more obfuscated. The avatars were just the first letter of our company name in our classic font, then generic clip art of people in tech support. The body became images instead of plain text. As we rolled out optical character recognition, the scammers began tweaking the resolution of the image and tinting the background to try to evade detection while still appearing legitimate enough to convince our sellers.

The most frustrating part of this work was that the scammers always seemed to have a new tactic that we would have to account for. The relentless evolution of their attacks brought us to a boiling point when one of my team members openly suggested we could have a mole in our company. That’s when the director overseeing the project brought us back down to Earth. “It’s simple,” he said, “these scammers are just using the QA mindset to find our weaknesses.”

While I’m not suggesting you start an illegal social engineering ring, the analogy is apt. These scammers were constantly asking, what if? What if I swap out the avatar? What if I darken the image background? What if I tweaked the language here? While their actions had a very specific goal of thwarting our defenses, the crux of their success was relentless curiosity. We can apply that same curiosity to our own testing practices to head off bad actors from doing it themselves.

My second example isn’t quite so bleak. Instead of using the QA mindset for nefarious purposes, this creator uses it for entertainment. Josh Knoles of the YouTube channel Let’s Game It Out regularly posts videos in which he gets an early-access video game and, for lack of a better description, intentionally plays it wrong. He forces his way out of bounds, pushes the limits on all features, and in general finds any way possible to complete his tasks while avoiding what the developer intended. As an example, here he is playing Parcel Simulator. I think this video is great. If you watch it, try to listen to Josh’s thought process as he goes through the game because it’s a fantastic model for how to approach an application with curiosity.

And that’s it! Four posts on the QA mindset. I hope you’ve gained some useful knowledge somewhere in all of this. I’ll be back again next time with a post on a new topic. In the meantime, feel free to keep the conversation going in the comments.

First off, let’s talk about three primary types of testing: regression, acceptance, and exploratory testing. Regression testing is when you know that a feature or flow has worked in the past and you are ensuring that it still works. Acceptance testing is when you are verifying that a new feature meets the needs outlined in its design. Exploratory testing is when you are trying out new flows that no one thought of before. This latter type once again puts you in the role of the tester from the three amigos planning session. Now that you have a functioning app in front of you, you seek out edge cases that no one considered.

While regression testing is incredibly important, it’s not reflective of the QA mindset. By the time you’ve gotten to this style of testing, you have a strong sense of what is supposed to happen. You might even have a complete test plan that you’ve already run through before. This style of testing can quickly become mindless box-checking that does not play to the strengths of a human. What’s more, regression testing also has quickly diminishing returns. The Pesticide Paradox tells us that the more times you run the same test, the less likely it is to catch a bug. All of these are reasons why we strive to offload regression testing to automated tools that are much cheaper than manual testing.

Acceptance testing seems like it would involve the QA mindset, but it’s possible to sidestep that mode of thinking while still verifying the feature. Consider the practice of QA parties. The act of writing a test plan offers us the opportunity to enter the QA mindset when identifying test cases. However, test cases can easily be pulled from the acceptance criteria of the feature, effectively just translating a set of requirements into a series of actions. Similarly, a developer going off-script during the QA party to try a new flow is an exploration, but QA parties frequently involve multiple developers running the same set of tests across different browsers, limiting the class of bugs they can catch. Furthermore, when the person writing the test plan is the same person that wrote the code, then the plan is susceptible to the same attentional bias we talked about in the last post.

So regression testing has a diminishing probability of catching bugs, and acceptance testing typically confirms what we already expected to be true. How do we create space to find the unexpected bug? Ultimately, the most valuable asset humans bring to testing is our novel thinking, and we need time to express that through exploratory testing. This helps ferret out difficult to anticipate failure conditions and creates an overall more polished product for our users. This is the power of the QA mindset. That’s why we need to intentionally carve out time for high-leverage exploratory testing as part of our standard process, and it’s the reason why I advocate for peer testing at the pull request stage.

That’s all for now. If you have any thoughts about creating space for testing, feel free to drop them in the thread or pop into the comments. Next post we’ll talk about the most difficult aspect of the QA mindset: how to bring curiosity to your testing. Until then!

You may be wondering, why is distance from the code so fundamental to the QA mindset? Speaking for myself, I was a QA / SDET for 7+ years. I am very capable of testing things well. However, the difference between when I’m testing my own code versus when I’m testing someone else’s code is night and day. If I just spent a day hammering on a problem, and I finally get it to work, then I often only spend my time on narrow acceptance testing. If I’m coming in fresh to a change I’ve never seen before, I’m much more capable of approaching the task with the curiosity necessary for effective exploratory testing.

So what accounts for this difference? I attribute this to a form of attentional bias. Attentional bias is the tendency to focus on a particular stimulus that grabs our attention or is somehow emotionally charged, at the expense of missing other, less gripping information. When we as developers have been hard at work writing code, we are more likely to focus on the satisfaction of finishing a difficult feature than the work of ferreting out bugs hidden in edge cases. This inhibits our ability to step into the QA mindset, and it’s why I always advocate that someone other than the code owner should test a change during the pull request stage. This process is not without its cost. The reviewer needs to fully context switch to review mode, often stashing their own in-flight changes to load up a feature branch in their development environment. However, the cost of not doing so is a test run with blinders on.

This is also why it’s so important to have someone play the role of tester in the planning phase. As I described in my last post, the tester is one of the three amigos of planning, and is tasked with thinking about edge cases. Their attention is intentionally drawn away from how the feature should work as they are encouraged to focus on how the feature will break. This can be a difficult role to step into at first, so let’s take a moment to talk about frameworks for planning and writing work tickets that can make this easier. I’m a proponent of the INVEST framework, which states a ticket should be:

• Independent

• Negotiable

• Valuable

• Estimable

• Small

• Testable

I like to apply this framework to any ticket that gets reviewed in refinement or team planning. For teams new to INVEST, I would even suggest explicitly asking these questions for each ticket. “Is it Independent? Is it Negotiable? Is it Valuable? …” As you build this muscle, discussions about story writing will naturally occur in planning. Whether you’re an expert in this process or giving it a try for the first time, you’ll likely find that asking whether a ticket is testable inevitably leads to questions of how you will test it and what you will test. This works wonders for improving feature quality. When we create space for these discussions and normalize this mode of thinking, we begin to expand our attention beyond the narrow criteria of the feature. We step into the QA mindset and see the broader space this work occupies in the whole system.

That’s it for this post. Next time we’ll pick up again with a discussion of carving out time for testing!

As the name suggests, the QA mindset is a state of mind that a person needs to enter in order to effectively test, well, anything really. It prepares you to look at a piece of software objectively so that you can identify if it is ready to be sent to our users. Stepping into the QA mindset is a skill unto itself, and interviews for Quality Assurance Engineers typically try to ferret out how well a candidate can leverage this mode of thinking.

Over the next few posts I’ll try to capture how I approached the QA mindset during the 7+ years when manual testing was a core function of my work. However, before we get into the what and how of the matter, it’s important to understand when this mode of thinking should be applied. The obvious answer is that you need it when you are testing a piece of software. While this is objectively correct, if you’re waiting until a story is dev complete to leverage the QA mindset, then you are likely days if not weeks behind the curve. Discussions about testing should at least happen during story refinement, and for larger initiatives they should take place during product design.

To understand what those conversations look like, it’s useful to refer to Cucumber and the tenets of Behavior-Driven Development. Many of you may have used Cucumber as a testing tool, and if so you probably hated it. There’s a simple reason for that: Cucumber is not a testing tool. As its creators have said, Cucumber is the world’s most misunderstood collaboration tool. Viewed through that lens, it’s no surprise that The Cucumber Book spends a significant amount of time discussing the practice of testing. This portion in particular has always stuck with me:

Three Amigos

The best [test] scenarios are created when the three amigos come together, bringing three very different perspectives:

- The first amigo is a tester, who thinks about how to break things. The tester will typically come up with lots of scenarios, sometimes covering obscure edge cases and sometimes covering very important ones that no one else had thought of.

- The second amigo is a programmer, who thinks about how to make things. The programmer will typically add steps to scenarios, as he asks clarifying questions about what exactly should happen.

- The third amigo is the product owner, who cares about scope. When the tester thinks of scenarios that hit edge cases the product owner doesn’t care about, she can tell the team to drop that scenario out of scope altogether, or the group can decide to test it with unit tests only. When the programmer explains that implementing a particular scenario will be complicated, the product owner has the authority to help decide on alternatives or to drop it altogether.

You may be wondering, what does that mean for my company that has not hired dedicated Quality Assurance Engineers to be the first amigo? It’s important to remember that “tester” is not a job description. It’s a role that must be filled on a team. If we do not hire engineers who specialize in that role, then the folks we have must develop the skills necessary to play it themselves. That means it’s incumbent upon all of us to develop our QA mindset.

Over the next few posts, I’ll be digging into the three key factors for effectively entering and leveraging the QA mindset: distance from the code, ample time to explore, and curiosity. Stay tuned!