Developing the QA Mindset: Ample Time to Explore
Hey folks, I’m back again with another installment of the QA mindset series. Today I wanted to talk about the importance of allotting ample time to explore. This is a tricky one because it’s easy to spend a bunch of time testing without ever entering the QA mindset. In order to understand the nuances of testing, let’s discuss how and why we should be manually testing our application.
First off, let’s talk about three primary types of testing: regression, acceptance, and exploratory testing. Regression testing is when you know that a feature or flow has worked in the past and you are ensuring that it still works. Acceptance testing is when you are verifying that a new feature meets the needs outlined in its design. Exploratory testing is when you are trying out new flows that no one thought of before. This latter type once again puts you in the role of the tester from the three amigos planning session. Now that you have a functioning app in front of you, you seek out edge cases that no one considered.
While regression testing is incredibly important, it’s not reflective of the QA mindset. By the time you’ve gotten to this style of testing, you have a strong sense of what is supposed to happen. You might even have a complete test plan that you’ve already run through before. This style of testing can quickly become mindless box-checking that does not play to the strengths of a human. What’s more, regression testing also has quickly diminishing returns. The Pesticide Paradox tells us that the more times you run the same test, the less likely it is to catch a bug. All of these are reasons why we strive to offload regression testing to automated tools that are much cheaper than manual testing.
Acceptance testing seems like it would involve the QA mindset, but it’s possible to sidestep that mode of thinking while still verifying the feature. Consider the practice of QA parties. The act of writing a test plan offers us the opportunity to enter the QA mindset when identifying test cases. However, test cases can easily be pulled from the acceptance criteria of the feature, effectively just translating a set of requirements into a series of actions. Similarly, a developer going off-script during the QA party to try a new flow is an exploration, but QA parties frequently involve multiple developers running the same set of tests across different browsers, limiting the class of bugs they can catch. Furthermore, when the person writing the test plan is the same person that wrote the code, then the plan is susceptible to the same attentional bias we talked about in the last post.
So regression testing has a diminishing probability of catching bugs, and acceptance testing typically confirms what we already expected to be true. How do we create space to find the unexpected bug? Ultimately, the most valuable asset humans bring to testing is our novel thinking, and we need time to express that through exploratory testing. This helps ferret out difficult to anticipate failure conditions and creates an overall more polished product for our users. This is the power of the QA mindset. That’s why we need to intentionally carve out time for high-leverage exploratory testing as part of our standard process, and it’s the reason why I advocate for peer testing at the pull request stage.
That’s all for now. If you have any thoughts about creating space for testing, feel free to drop them in the thread or pop into the comments. Next post we’ll talk about the most difficult aspect of the QA mindset: how to bring curiosity to your testing. Until then!
Would also like to add that an effective way to test any app is to give it to people that know nothing (or, well, very little) about what it's supposed to do. With them, you'll get more corner-case inputs and subtask step reordering which often is where new problems arise (and which may suggest new guard-rail requirements in the user experience). Effective use of any software requires a "conceptual model" in which standard use cases exist (and make sense). New users (prospective customers?) lack that well-honed model to shape their interaction with the app in the same manner as designers and developers approach it ... with interesting and valuable outcomes.
Yes, I agree with the presented triumvirate of testing types. I wanted to layer in, somewhere, stress testing. My first take is to put it in 'Exploratory'. A couple examples - I faced a task of testing staged releases of residential CableTV set-top-box software (eg, the DVR features running inside your Cable box, as well as simple channel navigation, etc.). Adopting the QA Mindset, I first imagined "the household pre-teen" sitting on the Cable box remote control, mindlessly, and thus sending a stream of hundreds (thousands?) of "Channel Up" button presses (pick your poison here - volume button, whatever). How does the Cable box software respond to thousands of consecutive Channel Up messages? Next, I thought - well, we can schedule a DVR recording just fine. Can we schedule 10 or 20 of them, each one minute in length? And what if we intersperse Cancel Record interactions during the period in which the box is recording broadcast(s)? In each case, my tests found bugs (right away - system crashes) that I suspect the developers had not personally encountered. Sometimes, the designer and developer are challenged to identify "never intended" ways of interacting with the system. This is where new personnel can add value to the process.