The first example comes from a job many years ago. I worked at a two-sided market place, so our users were both people buying things and people selling things. The sellers were a particularly lucrative target for scammers. The ploy was simple: a scammer would create an account with an avatar and name that appeared to be a site admin, send the seller a message that a sale had failed to process, and ask the seller to provide updated bank account information. I was on a project that was trying to prevent those scammers from contacting our sellers.

The messages started off easy to identify. The body was plain text, the username was overt, the avatar was simply our company logo. We built an AI model that classified these as spam, and the rate of sellers opening these messages dropped precipitously. In response, the scam messages steadily became more obfuscated. The avatars were just the first letter of our company name in our classic font, then generic clip art of people in tech support. The body became images instead of plain text. As we rolled out optical character recognition, the scammers began tweaking the resolution of the image and tinting the background to try to evade detection while still appearing legitimate enough to convince our sellers.

The most frustrating part of this work was that the scammers always seemed to have a new tactic that we would have to account for. The relentless evolution of their attacks brought us to a boiling point when one of my team members openly suggested we could have a mole in our company. That’s when the director overseeing the project brought us back down to Earth. “It’s simple,” he said, “these scammers are just using the QA mindset to find our weaknesses.”

While I’m not suggesting you start an illegal social engineering ring, the analogy is apt. These scammers were constantly asking, what if? What if I swap out the avatar? What if I darken the image background? What if I tweaked the language here? While their actions had a very specific goal of thwarting our defenses, the crux of their success was relentless curiosity. We can apply that same curiosity to our own testing practices to head off bad actors from doing it themselves.

My second example isn’t quite so bleak. Instead of using the QA mindset for nefarious purposes, this creator uses it for entertainment. Josh Knoles of the YouTube channel Let’s Game It Out regularly posts videos in which he gets an early-access video game and, for lack of a better description, intentionally plays it wrong. He forces his way out of bounds, pushes the limits on all features, and in general finds any way possible to complete his tasks while avoiding what the developer intended. As an example, here he is playing Parcel Simulator. I think this video is great. If you watch it, try to listen to Josh’s thought process as he goes through the game because it’s a fantastic model for how to approach an application with curiosity.

And that’s it! Four posts on the QA mindset. I hope you’ve gained some useful knowledge somewhere in all of this. I’ll be back again next time with a post on a new topic. In the meantime, feel free to keep the conversation going in the comments.

First off, let’s talk about three primary types of testing: regression, acceptance, and exploratory testing. Regression testing is when you know that a feature or flow has worked in the past and you are ensuring that it still works. Acceptance testing is when you are verifying that a new feature meets the needs outlined in its design. Exploratory testing is when you are trying out new flows that no one thought of before. This latter type once again puts you in the role of the tester from the three amigos planning session. Now that you have a functioning app in front of you, you seek out edge cases that no one considered.

While regression testing is incredibly important, it’s not reflective of the QA mindset. By the time you’ve gotten to this style of testing, you have a strong sense of what is supposed to happen. You might even have a complete test plan that you’ve already run through before. This style of testing can quickly become mindless box-checking that does not play to the strengths of a human. What’s more, regression testing also has quickly diminishing returns. The Pesticide Paradox tells us that the more times you run the same test, the less likely it is to catch a bug. All of these are reasons why we strive to offload regression testing to automated tools that are much cheaper than manual testing.

Acceptance testing seems like it would involve the QA mindset, but it’s possible to sidestep that mode of thinking while still verifying the feature. Consider the practice of QA parties. The act of writing a test plan offers us the opportunity to enter the QA mindset when identifying test cases. However, test cases can easily be pulled from the acceptance criteria of the feature, effectively just translating a set of requirements into a series of actions. Similarly, a developer going off-script during the QA party to try a new flow is an exploration, but QA parties frequently involve multiple developers running the same set of tests across different browsers, limiting the class of bugs they can catch. Furthermore, when the person writing the test plan is the same person that wrote the code, then the plan is susceptible to the same attentional bias we talked about in the last post.

So regression testing has a diminishing probability of catching bugs, and acceptance testing typically confirms what we already expected to be true. How do we create space to find the unexpected bug? Ultimately, the most valuable asset humans bring to testing is our novel thinking, and we need time to express that through exploratory testing. This helps ferret out difficult to anticipate failure conditions and creates an overall more polished product for our users. This is the power of the QA mindset. That’s why we need to intentionally carve out time for high-leverage exploratory testing as part of our standard process, and it’s the reason why I advocate for peer testing at the pull request stage.

That’s all for now. If you have any thoughts about creating space for testing, feel free to drop them in the thread or pop into the comments. Next post we’ll talk about the most difficult aspect of the QA mindset: how to bring curiosity to your testing. Until then!

You may be wondering, why is distance from the code so fundamental to the QA mindset? Speaking for myself, I was a QA / SDET for 7+ years. I am very capable of testing things well. However, the difference between when I’m testing my own code versus when I’m testing someone else’s code is night and day. If I just spent a day hammering on a problem, and I finally get it to work, then I often only spend my time on narrow acceptance testing. If I’m coming in fresh to a change I’ve never seen before, I’m much more capable of approaching the task with the curiosity necessary for effective exploratory testing.

So what accounts for this difference? I attribute this to a form of attentional bias. Attentional bias is the tendency to focus on a particular stimulus that grabs our attention or is somehow emotionally charged, at the expense of missing other, less gripping information. When we as developers have been hard at work writing code, we are more likely to focus on the satisfaction of finishing a difficult feature than the work of ferreting out bugs hidden in edge cases. This inhibits our ability to step into the QA mindset, and it’s why I always advocate that someone other than the code owner should test a change during the pull request stage. This process is not without its cost. The reviewer needs to fully context switch to review mode, often stashing their own in-flight changes to load up a feature branch in their development environment. However, the cost of not doing so is a test run with blinders on.

This is also why it’s so important to have someone play the role of tester in the planning phase. As I described in my last post, the tester is one of the three amigos of planning, and is tasked with thinking about edge cases. Their attention is intentionally drawn away from how the feature should work as they are encouraged to focus on how the feature will break. This can be a difficult role to step into at first, so let’s take a moment to talk about frameworks for planning and writing work tickets that can make this easier. I’m a proponent of the INVEST framework, which states a ticket should be:

• Independent

• Negotiable

• Valuable

• Estimable

• Small

• Testable

I like to apply this framework to any ticket that gets reviewed in refinement or team planning. For teams new to INVEST, I would even suggest explicitly asking these questions for each ticket. “Is it Independent? Is it Negotiable? Is it Valuable? …” As you build this muscle, discussions about story writing will naturally occur in planning. Whether you’re an expert in this process or giving it a try for the first time, you’ll likely find that asking whether a ticket is testable inevitably leads to questions of how you will test it and what you will test. This works wonders for improving feature quality. When we create space for these discussions and normalize this mode of thinking, we begin to expand our attention beyond the narrow criteria of the feature. We step into the QA mindset and see the broader space this work occupies in the whole system.

That’s it for this post. Next time we’ll pick up again with a discussion of carving out time for testing!

As the name suggests, the QA mindset is a state of mind that a person needs to enter in order to effectively test, well, anything really. It prepares you to look at a piece of software objectively so that you can identify if it is ready to be sent to our users. Stepping into the QA mindset is a skill unto itself, and interviews for Quality Assurance Engineers typically try to ferret out how well a candidate can leverage this mode of thinking.

Over the next few posts I’ll try to capture how I approached the QA mindset during the 7+ years when manual testing was a core function of my work. However, before we get into the what and how of the matter, it’s important to understand when this mode of thinking should be applied. The obvious answer is that you need it when you are testing a piece of software. While this is objectively correct, if you’re waiting until a story is dev complete to leverage the QA mindset, then you are likely days if not weeks behind the curve. Discussions about testing should at least happen during story refinement, and for larger initiatives they should take place during product design.

To understand what those conversations look like, it’s useful to refer to Cucumber and the tenets of Behavior-Driven Development. Many of you may have used Cucumber as a testing tool, and if so you probably hated it. There’s a simple reason for that: Cucumber is not a testing tool. As its creators have said, Cucumber is the world’s most misunderstood collaboration tool. Viewed through that lens, it’s no surprise that The Cucumber Book spends a significant amount of time discussing the practice of testing. This portion in particular has always stuck with me:

Three Amigos

The best [test] scenarios are created when the three amigos come together, bringing three very different perspectives:

- The first amigo is a tester, who thinks about how to break things. The tester will typically come up with lots of scenarios, sometimes covering obscure edge cases and sometimes covering very important ones that no one else had thought of.

- The second amigo is a programmer, who thinks about how to make things. The programmer will typically add steps to scenarios, as he asks clarifying questions about what exactly should happen.

- The third amigo is the product owner, who cares about scope. When the tester thinks of scenarios that hit edge cases the product owner doesn’t care about, she can tell the team to drop that scenario out of scope altogether, or the group can decide to test it with unit tests only. When the programmer explains that implementing a particular scenario will be complicated, the product owner has the authority to help decide on alternatives or to drop it altogether.

You may be wondering, what does that mean for my company that has not hired dedicated Quality Assurance Engineers to be the first amigo? It’s important to remember that “tester” is not a job description. It’s a role that must be filled on a team. If we do not hire engineers who specialize in that role, then the folks we have must develop the skills necessary to play it themselves. That means it’s incumbent upon all of us to develop our QA mindset.

Over the next few posts, I’ll be digging into the three key factors for effectively entering and leveraging the QA mindset: distance from the code, ample time to explore, and curiosity. Stay tuned!