Teaching AI To Test: The Process

Hey folks, in my last post I spoke about why the return on AI investment is low when you only incorporate agents into the coding portion of software development. In this post, I’ll be digging into practical ways to enable agents to perform higher order testing. For this blog I’ll be focusing on end-to-end style testing of a web application, but the basic principles are applicable to other fields of software development.

First of all, let’s talk about the technical setup. In the realm of web development the highest order testing is loading the application in an actual browser. You can quickly bootstrap a testing harness for this using Claude Code, Playwright, and the playwright-cli binary / Claude skill. If you also install Xvfb then this whole setup can run headlessly on a virtual environment, allowing you to manage work on multiple features at once.

Now that we have the tech sorted out we should talk about what we’re trying to accomplish. Let’s say our goal is to enable AI to work on features fully autonomously. With a single command we want an agent to complete the following steps:

1. Review the work ticket, associated designs, and relevant areas of the code.

2. Make the code changes necessary to complete the ticket.

3. Execute user acceptance testing.

4. Make any additional changes based on the testing outcome.

5. Record itself running a final verification of the code changes.

6. Open a pull request for human review.

7. Post the verification records to the pull request.

Steps 3 and 4 are intended to improve the quality of the change set prior to human review to limit the amount of time we sink into correcting the agent. Additionally, steps 5 and 7 ensure the human receives proof in the pull request that satisfactory testing was completed. For our web app example this takes the form of screenshots / screencasts from the latest testing attempt that demonstrate the feature meets the acceptance criteria described in the ticket.

So how do we give the agent the context necessary to perform that testing? You might be inclined to open a text editor and write page after page of instructions about how your feature works and the best ways to test it. Resist that urge. Instead, boot up Claude Code, ask it to launch Playwright, and instruct it to navigate to your application. Give it a guided exploration of your area of the codebase. For instance, try sending it prompts like these:

What do you think the purpose of this page is?

Notice there’s a button with a gear icon in the corner. What does that do?

Try switching to a different mode. How does that change the experience?

The following is the URL for our admin panel. Notice that is has the capability to reset metered features. How could that be helpful for testing?

Once the agent has a sense of how the app works at a high level, we can introduce it to some of the deeper inner workings like the database. After all, being able to insert a test account into a database is much faster and less error-prone than registering one through the UI. Connect Claude Code to your codebase and a development database—not your Production database—and explore the schemas and ORMs that underly your test data:

Inspect the contents of the Users table. Notice there are different types of users. How would that come in to play when testing a feature?

The following is the ORM definition for a User. How could you use that to insert a new user into the database?

Now that you’ve spent some time exploring your feature, ask the agent that did the exploration to write the skill for you:

We have a lot of useful knowledge about this feature. Write a new skill that captures all of that knowledge so future agents can use it as well.

Working through this process will give the agent a great start for testing your feature, but remember that setting up high-level test harnesses requires time and effort. Teaching an agent to test your application is no different. Many developers find Selenium-style tests hard to write compared to unit tests because Selenium-style tests are running on an environment that behaves in a non-deterministic way. Autonomous agent testing compounds the problem as the agent itself behaves in a non-deterministic way:

• Unit Tests: Deterministic tool, deterministic environment

• Selenium: Deterministic tool, non-deterministic environment

• Autonomous Agents: Non-deterministic tool, non-deterministic environment

This means the best advice for writing solid Selenium tests also applies to writing solid autonomous agent testing skills. You must run them over and over again to see what breaks. Concretely, follow these steps:

1. Identify a ticket that would require your new skill to test.

2. Have an agent write the code for that ticket, but have it pause at the testing stage.

3. Commit the code change from Step 2 to a branch.

4. Spin up a brand new development environment and launch a new Claude Code agent to ensure you’re starting with a clean context.

5. Check out the branch from Step 3.

6. Tell your agent it is on a branch with untested code changes for a particular ticket. Ask it to verify the change.

7. Observe the agent’s behavior, particularly what goes wrong.

   • Does it get confused when it sees a 500 responses from ancillary services?

   • Does it get stuck in a loop because it missed an error message?

   • When something goes wrong, does the agent have a useful fallback strategy to complete its testing?

8. Repeat Steps 4 through 7. Ideally run through this process five to ten times.

The more you battle test your skills against actual agent-generated code in actual development environments, the more self-sufficient future agents will be in their testing process.

With that, we now have a process to teach an agent how to test your application. In the next post I’ll dig into practical tips for organizing these instructions so they are understandable and actionable by agents.