"QA"

It’s like there aren’t enough testing-related terms out there so people have to just use the word “QA” to talk about anything in the testing domain.

  • “Ask QA to test this.” (It’s a group of people)
  • “QA is down again!” (It’s a test environment)
  • “We’ll need someone to QA this before it goes out.” (It’s an official action)
  • “Is it in QA”? (You’re either asking if someone is testing it or you’re asking if code resides in a specific environment)
...sigh

By speaking properly, ourselves, perhaps we can change this.

  • “Ask the testers to test this.”
  • “Environment [insert an environment name, don't name it "QA"] is down again!”
  • “We’ll need someone to test this before it goes out.”
  • "Is it being tested by the testers?"

If you read my previous post, you know I help test a product on iteration 80. This product has few automated regression tests (we are working to change that). Currently, my test team faces regression testing armed with nothing more than their brains, their hands, and a big ticking clock.

Each 4 week iteration provides us with a little less than a week to spend regression testing. If you can relate to my situation, you may be interested in two painless practices we have adopted to better handle our ever increasing challenge of regression testing.

  1. Group Regression Test Planning – We get the devs, BAs, and testers into a room and spend an hour brainstorming, to determine the top priority regression test areas of focus. We scan through the iteration's changes and draw on each other’s knowledge to create a draft of what will become the priority “1” tests or areas of focus. This occurs on a shared MS Excel regression test list spreadsheet. The prior iteration’s regression tests drop in priority...and so on.

  2. Group Regression Test Sessions – We fill a large bowl with chocolate and invite the BAs and devs to join the testers in our classroom for two 3-hour sessions of group regression testing. All participants track their progress on the shared regression test list and we tackle it as a team. The team approach also occasionally shakes out multi-user-scenario-bugs, since our product is heavily dependent on user locking.

Our approach is not to complete a set amount of regression tests before we ship. Instead, we complete as many tests as we can, within the time provided. The prioritized test list makes this possible and shields us from things outside of our control.

Is anyone else doing regression testing with mostly manual testing? How do you pull it off?

Testing updates to old software is much more difficult than testing updates to new software. The older it gets, the more cobwebs form in lonely corners, which drop out of test rotation and concern.

One of my software projects is on its 5th year of steady updates (iteration 80). In software years, this product is 80-years-old. It makes me think of my house, which happens to turn 80-years-old this year.

Seemingly easy upgrades to my old house often result in way more changes than initially planned. When my wife talked me into replacing the cracked pink bathroom tiles, I soon uncovered 6” deep concrete poured between joists that had all but rotted through (due to 79 years of direct contact with concrete.


In the 1920's, I’m sure floating concrete floors were an excellent choice for tiled floors. And fortunately, the old growth pine joists were strong enough to hold the weight…but not for 80 years.

One thing leads to another and before we knew it, we were rebuilding our entire bathroom, including the floor.


Then the changes start to expand way out of scope. The walls will not be compatible with the new floor, so those will have to be replaced (but maybe we'll just patch and replace the worst parts of the outside wall).

Then you realize older technologies are not compatible with newer. You mean Home Depot doesn’t sell Quest plumbing fittings anymore? You mean I have to replace my polybutylene pipes with copper just to move my toilet?

Meanwhile, back at work, my users were asking for the seemingly easy upgrade to allow filtering in the product’s massive control-center-like grid. Sure, no problem…oh, hold on…we can do it but the current grid won’t support it so we’ll have to rebuild the entire module, a testing nightmare (depending on your perspective).

So be careful with work estimates on seemingly simple changes to older products. And try to remember all the often ignored, dependent systems that are affected.

Yesterday a manager told me they were troubled by a showstopper bug that was found after my team had finished testing a particular build.

(My team uses the “Time’s Up!” heuristic to determine when we’re done testing.)

Then the manager asked me what I could do to prevent this from happening again. I thought long and hard on this question and just when I was about to concede defeat, the ultimate solution came to me.

It may take several iterations, but if my team can build a time machine, we can guarantee no showstopper bugs will escape. I think it will really shorten our test cycles too. We’ll just throw stuff to the users, see what problems they encounter, then travel back in time and log it as a Showstopper...or maybe we'll call it a "Show Stopped". Better yet, we’ll just explain it to dev before they write it.

I just have to work out the predestination paradox, so I don’t get stuck in a causal loop, eternally traveling back in time to prevent the same bug from occurring.


My favorite recent time travel movies:

Your application under test (AUT) probably interfaces with external systems. Fact: these external systems will go down at some point while a user is attempting to use your AUT.

Here is the obvious test:

  1. Take ExternalSystemA down. If this is outside your control, simulate ExternalSystemA’s outage by changing where your AUT points for ExternalSystemA.
  2. Trigger whatever user operations cause your AUT to interface with ExternalSystemA.
Expected Results: The user gets a friendly message indicating some functionality is blocked at this time. The support team gets an error alert indicating ExternalSystemA is not responding.



We executed the above test for 6 or 7 external systems and got our AUT robust enough to only block minimum functionality and provide good communication to users and the support team. However, just when we were getting cocky, we encountered a slight variation on the first test that crippled our AUT. Here is the test we missed for each external system.

  1. Put ExternalSystemA into a state where it is up and running but cannot respond to your AUT within the amount of time your AUT is willing to wait. Note: We were able to simulate this by taking down ExternalSystemB, which gets called by ExternalSystemA.
  2. Trigger whatever user operations cause your AUT to interface with ExternalSystemA.
Expected Results: The user gets a friendly message indicating some functionality is blocked at this time. The support team gets an error alert indicating ExternalSystemB is not responding.

Question: What metric or easily understood information can my test team provide users, to show our contribution to the software we release?

I just got back from vacation and am looking at a beautiful pie chart that shows the following per iteration:

  • # of features delivered
  • # of bugs found in test vs. prod
  • # of bugs fixed
  • # of test cases executed
After a series of buggy production releases, my team (or at least the BAs) have decided to provide users with colorful charts depicting how hard we’ve been working each iteration. My main gripe is providing my BAs with a # representing executed test cases.

First, I feel uncomfortable measuring tester value based on test case count, for obvious reasons.

Second, the pie chart looks like all we do is test. One slice lists 400 tests. Another lists 13 features...strange juxtaposition.

Third, I’m not even sure how to provide said count. I certainly don’t encourage my test team to exhaustively document their manual test cases, nor do I care how many artifacts they save distinct tests within. Do I include 900+ automated UI test executions? Do I include thousands more unit test executions? Does the final # speak to users about quality? Does it represent how effective testers are? Not to me. Maybe it does to users...

PR is important, especially when your reputation takes a dive. I, too, want to show the users how hard my QA team works. I want to show it in the easiest possible way. I could provide a long list of tests, but they don't want to read that. What am I missing? What metric or easily understood information can my test team provide users, to show our contribution to the software we release?

We have a new mantra on my team.

“Don’t fix bugs unless users want them fixed.”

When testers found bugs that were already in production, we used to just fix them. Soon we realized, fixing them may do more harm than good. For example, we have a time control in one of our apps that accepts input in a format like HH:MM:SS. We noticed inconsistent behavior across instances of these controls in the app; things like some control instances would force leading zero while others would not, some would allow double-clicking-to-select time units while others would not.

We logged a bug to standardize these time controls throughout our app. When the bits went to prod the users screamed bloody murder. They hated the change and complained about disruptions. It turns out the users didn’t even know the inconsistency existed in the first place. As devs, BAs, and testers, we’re in and out of said time controls all over the app. But in production, users tend to only work in one of about 10 different modules based on their jobs. They could care less how the time control worked in neighboring modules.

“Don’t fix bugs unless users want them fixed.”

This mantra also applies to larger problems testers find. A room full of devs and testers can pat themselves on the back, thinking users will love them for certain bug fixes, only to find the users had adjusted to the broken code and want it back. And the danger increases the longer your app has been in production.

What’s your definition of a bug? My favorite is one I learned at Michael Bolton’s Rapid Software Testing class.

A bug is something that bugs someone who matters.

Keep that in mind the next time your team starts fixing bugs the users haven’t complained about.



Copyright 2006| Blogger Templates by GeckoandFly modified and converted to Blogger Beta by Blogcrowds.
No part of the content or the blog may be reproduced without prior written permission.