…is the checks, stupid.

It doesn’t matter what framework you are using, what language you write them in, or how many you have.  What matters is how effectively your automated checks help determine ship decisions.

1. What should your automated checks observe?
2. What decision rules should your automated checks use to determine pass/fail? 

Those are the two hardest questions to answer.  You can’t Google them.  You can’t ask your testing mentors.  You’re tempted to hide your decisions when they’re poorly conceived (because you know few will ask).  You’re tempted to focus on what you know people will ask; the questions with the shortest answers:  How many automated checks?  Did they pass?

But, what are they checking?  You know that’s what matters.  Start building your automated check suite there.  The rest can follow.

In reference to my When Do We Need Detailed Test Cases? post, Roshni Prince asked:

“when we run multiple tests in our head… [without using detailed test cases] …how can we be really sure that we tested everything on the product by the end of the test cycle?”

Nice question, Roshni.  I have two answers.  The first takes your question literally.

• …We can’t.  We’ll never test everything by the end of the test cycle.  Heck, we’ll never test everything in an open-ended test cycle.  But who cares?  That’s not our goal.
• Now I’ll answer what I think you are really asking, which is “without detailed test cases, how can we be sure of our test coverage?”.  We can’t be sure, but IMO, we can get close enough using one or more of the following approaches:
1. Write “test ideas” (AKA test case fragments).  These should be less than the size of a Tweet.  These are faster than detailed test cases to write/read/execute and more flexible.
2. Use Code Coverage software to visually analyze test coverage.
3. Build a test matrix using Excel or another table.
4. Use a mind map to write test ideas.  Attach it to your specs for an artifact.
5. Use a Session Based Test Management tool like Rapid Reporter to record test notes as you test.
6. Use a natural method of documenting test coverage.  By “natural” we mean, something that will not add extra administrative work.  Regulatory compliance expert and tester, Griffin Jones, has used audio and/or video recordings of test sessions to pass rigorous audits.  He burns these to DVD and has rock solid coverage information without the need for detailed test cases.  Another approach is to use keystroke capture software.
7. Finally, my favorite when circumstances allow; just remember!  That’s right, just use your brain to remember what you tested.  Brains rock!  Brains are so underrated by our profession.  This approach may help you shine when people are more interested in getting test results quickly and you only need to answer questions about what you tested in the immediate future…like today!  IMO, the more you enjoy your work as a tester, the more you practice testing, the more you describe your tests to others, the better you’ll recall test coverage from your brain.  And brains record way more than any detailed test cases could ever hope to.

In my Don’t Give Test Cases To N00bs post I tried to make the argument against writing test cases as a means to coaching new testers.  At the risk of sounding like a test case hater, I would like to suggest three contexts that may benefit from detailed test cases.

These contexts do not include the case of a mandate (e.g., the stakeholder requires detailed test cases and you have no choice).

1. Automated Check Design: Whether a sapient tester is designing an automated check for an automation engineer or an automation engineer is designing the automated check herself, detailed test cases may be a good idea.  Writing detailed test cases will force tough decisions to be made prior to coding the check.  Decisions like: How will I know if this check passes?  How will I ensure this check’s dependent data exists?  What state can I expect the product-under-test to be in before the check’s first action?
2. Complex Business Process Flows:  If your product-under-test supports multiple ways of accomplishing each step in its business process flows, you may want to spec out each test to keep track of test coverage.  Example: Your product’s process to buy a new widget requires 3 steps.  Each of the 3 steps has 10 options.  Test1 may be: perform Step1 with Option4, perform Step2 with Option1, then perform Step3 with Option10.
3. Bug Report Repro Steps: Give those programmers the exact foot prints to follow else they’ll reply, “works on my box”.

Those are the three contexts I write detailed test cases for.  What about you?

In response to my What I Love About Kanban As A Tester #1 post, Anonymous stated:

“The whole purpose of documenting test cases…[is]…to be able to run [them] by testers who don’t have required knowledge of the functionality.”

Yeah, that’s what most of my prior test managers told me, too…

“if a new tester has to take over your testing responsibilities, they’ll need test cases”

I wouldn’t be surprised if a secret QA manager handbook went out to all QA managers, stating the above as the paramount purpose of test cases.  It was only recently that I came to understand how wrong all those managers were.

Before I go on, let me clarify what I mean by “test cases”.  When I say “test cases”, I’m talking about something with steps, like this:

1. Drag ItemA from the catalog screen to the new order screen.
2. Change the item quantity to “3” on the new order screen.
3. Click the “Submit Order” button.

Here’s where I go on: 

• When test cases sit around, they get stale.  Everything changes…except your test cases.  Giving these to n00bs is likely to result in false fails (and maybe even rejected bug reports).
• When test cases are blindly followed, we miss the house burning down right next to the house that just passed our inspection.
• When test cases are followed, we are only doing confirmatory testing.  Even negative (AKA “unhappy”) paths are confirmatory testing.  If that’s all we can do, we are one step closer to shutting down our careers as testers.
• Testing is waaaay more than following steps.  To channel Bolton, a test is something that goes on in your brain.  Testing is more than answering the question, “pass or fail?”.  Testing is sometimes answering the question, “Is there a problem here?”. 
• If our project mandates that testers follow test cases, for Pete’s sake, let the n00b’s write their own test cases.  It may force them to learn the domain.
• Along with test cases comes administrative work.  Perhaps time is better spent testing.
• If the goal is valuable testing from the n00b, wouldn’t that best be achieved by the lead tester coaching the n00b? And if that lead tester didn’t have to write test cases for a hypothetical n00b, wouldn’t that lead tester have more time to coach the hypothetical n00b, should she appear.  Here’s a secret: she never will appear.  You will have a stack of test cases that nobody cares about; not even your manager.

In my next post I’ll tell you when test cases might be a good idea.

...regression testing is optional.  What?  The horror!!!

Back in the dark ages, with Scrum, we were spending about 4 days of each iteration regression testing.  This product has lots of business logic in the UI, lots of drag-and-drop-type functionality, and very dynamic data, so it has never been a good candidate for automation.  Our regression test approach was to throw a bunch of humans at it (see my Group Regression Testing and Chocolate post).  With Scrum, each prod deployment was a full build, including about 14 modules, because lots of stuff got touched.  Thus, we always did a full regression test, touching all the modules.  Even after an exhaustive regression test, we generally had one or two “escapes” (i.e., bugs that escaped into production).

Now, ask me how often regression tests failed?  …not very often.  And, IMO, that is a lot of waste.

With Kanban, each prod release only touches one small part of the product.  So we are no longer doing full builds.  Think of it like doing a production patch.  We’ve gotten away from full regression tests because, with each release, we are only changing one tiny part of the product.  It is far less risky.  Why test the hell out of bits that didn’t change? 

So now we regression test based on one risk: the feature going to prod.  Sometimes it means an hour of regression tests.  Sometimes it means no regression tests.  So far, it’s a net loss of time spent on regression tests.  And this is a good thing.

We switched to Kanban in February.  So far, not a single escape has made it to prod (yes, I’m knocking on wood).

This success may just be a coincidence.  Or…maybe it’s easier for teams to prevent escaped bugs when those teams can focus on one Feature at a time.   Hmmmmmm…

For those of you writing automated checks and giving scrum reports, status reports, test reports, or some other form of communication to your team, please watch your language…and I'm not talking about swearing.

You may not want to say, “I found a bunch of issues”, because sometimes when you say that, what you really mean is, “I found a bunch of issues in my automated check code” or “I found a bunch of issues in our product code”.  Please be specific.  There is a big difference and we may be assuming the wrong thing.

If you often do checking by writing automated checks, you may not want to say, “I’m working on FeatureA”, because what you really mean is “I’m writing the automated checks for FeatureA and I haven't executed them or learned anything about how FeatureA works yet” or “I’m testing FeatureA with the help of automated checks and so far I have discovered the following…”

The goal of writing automated checks is to interrogate the system under test (SUT), right?  The goal is not just to have a bunch of automated checks.  See the difference?

Although your team may be interested in your progress creating the automated checks, they are probably more interested in what the automated checks have helped you discover about the SUT.

It’s the testing, stupid.  That’s why we hired you instead of another programmer.

...no testing deadlines…the freedom to test as long as I want.

Back in the dark ages, with Scrum, all the sprint Features had to be tested by the end of the iteration.  Since programming generally continued until that last minute (we couldn’t help ourselves), testers were sometimes forced to cut corners.  Even in cases where the whole team (e.g., programmers, BAs) jumped in to help test, there was still a tendency to skimp on testing that would otherwise be performed.  The team wants to be successful.  Success is more easily measured by delivered Features than Feature quality.  That’s the downside of deadlines.

With Kanban, there are no deadlines…you heard me!  Testers take as long as they need.  If the estimates are way off, it doesn’t leave gaps or crunches in iterations.  There are no iterations!  Warning: I fear unskilled testers may actually have a difficult time with this freedom.  Most testers are used to being told how much time they have (i.e., “The Time’s Up! Heuristic”).  So with Kanban, other Stopping Heuristics may become more important.