Last week I started testing an update to a complex legacy process.  At first, my head was spinning (it still kind of is).    There are so many inputs and test scenarios...so much I don’t understand.  Where to begin?

I think doing something half-baked now is better than doing something fully-baked later.  If we start planning a rigorous test based on too many assumptions we may not understand what we’re observing. 

In my case, I started with the easiest tests I could think of:

• Can I trigger the process-under-test? 
• Can I tell when the process-under-test completes?
• Can I access any internal error/success logging for said process?
• If I repeat the process-under-test multiple times, are the results consistent?

If there were a spectrum that showed a focus between learning by not manipulating and learning by manipulating something-under-test, it might look like this:

My tests started on the left side of the spectrum and worked right.  Now that I can get consistent results, let me see if I can manipulate it and predict its results: 

• If I pass ValueA to InputA, do the results match my expectations?
• If I remove ValueA from InputA, do the results return as before?
• If I pass ValueB to InputA, do the results match my expectations?

As long as my model of the process-under-test matches my above observations, I can start expanding complexity:

• If I pass ValueA and ValueB to InputA and ValueC and ValueD to InputB, do the results match my expectations?
• etc.

Now I have something valuable to discuss with the programmer or product owner.  “I’ve done the above tests.  What else can you think of?”.  It’s much easier to have this conversation when you’re not completely green, when you can show some effort.  It’s easier for the programmer or product owner to help when you lead them into the zone.

That worst is over.  The rest is easy.  Now you can really start testing!

Sometimes you just have to do something to get going.  Even if it’s half-baked.

About five years ago, my tester friend, Alex Kell, blew my mind by cockily declaring, “Why would you ever log a bug?  Just send the Story back.”

Okay.

My dev team uses a Kanban board that includes “In Testing” and “In Development” columns.  Sometimes bug reports are created against Stories.  But other times Stories are just sent left; For example, a Story “In Testing” may have its status changed to “In Development”, like Alex Kell’s maneuver above.  This normally is done using the Dead Horse When-To-Stop-A-Test Heuristic. We could also send an “In Development” story left if we decide the business rules need to be firmed up before coding can continue.

So how does one know when to log a bug report vs. send it left?

I proposed the following heuristic to my team today:

If the Acceptance Test Criteria (listed on the Story card) is violated, send it left.  It seems to me, logging a bug report for something already stated in the Story (e.g., Feature, Work Item, Spec) is mostly a waste of time.

Thoughts?

If you made two lists for a given software feature (or user story):

1. all the plausible user scenarios you could think of
2. all the implausible user scenarios you could think of

…which list would be longer?

I’m going to say the latter.  The user launches the product, holds down all the keys on the keyboard for four months, removes all the fonts from their OS, then attempts to save a value at the exact same time as one million other users.  One can determine implausible user scenarios without obtaining domain knowledge.

Plausible scenarios should be easier to predict, by definition.  It may be that only one out of 100 users stray from the “happy path”, in which case our product may have just experienced an implausible scenario.

What does this have to do with testing?  As time becomes dearer, I continue to refine my test approach.  It seems to me, the best tests to start with are still confirmatory (some call these “happy path”) tests.  There are fewer of them, which makes it more natural to know when to start executing the tests for the scenarios less likely to occur.

The chart above is my attempt to illustrate the test approach model I have in my head.  The Y axis is how plausible the test is (e.g., it is 100% likely that users will do this, it is 50% likely that users will do this).  The X axis represents the test order (e.g., 1st test executed, 2nd test executed, etc.).  The number of tests executed is relative.

Basically, I start with the most plausible tests, then shift my focus to the stuff that will rarely happen.  These rare scenarios at the bottom of the chart above can continue forever as you move toward 0% plausibility, so I generally use the “Times Up” stopping heuristic.  One can better tackle testing challenges with this model if one makes an effort to determine how users normally use the product.

I often hear contradictory voices in my head saying, “don’t start with confirmatory tests, the bugs are off the beaten path”.  Okay, but are they really?  If our definition of a bug is “something that bugs someone who matters”, then the problems I find on the bottom of the above chart’s line, may matter less than those found on the top.  Someone who matters, may not venture to the bottom. 

For more on my thoughts (and contrary thoughts) on this position see We Test To Find Out If Software *Can* Work.

After RST class (see my Four Day With Michael Bolton post), Bolton did a short critical thinking for testers workshop.  If you get an opportunity to attend one of these at a conference or other place, it’s time well spent.  The exercises were great, but I won’t blog about them because I don’t want to give them away.  Here is what I found in my notes…

• There are two types of thinking:
  1. System 1 Thinking – You use it all the time to make quick answers.  It works fine as long as things are not complex.
  2. System 2 Thinking – This thinking is lazy, you have to wake it up.
• If you want to be excellent at testing, you need to use System 2 Thinking.  Testing is not a straight forward technical problem because we are creating stuff that is largely invisible.
• Don’t plan or execute tests until you obtain context about the test mission.
• Leaping to assumptions carries risk.  Don’t build a network of assumptions.
• Avoid assumptions when:
  • critical things depend on it
  • when the assumption is unlikely to be true
  • the assumption is dangerous when not declared
• Huh?  Really?  So?   (James Bach’s critical thinking heuristic)
  • Huh? – Do I really understand?
  • Really? – How do I know what you say is true?
  • So? – Is that the only solution?
• “Rule of Three” – If you haven't thought of at least three plausible explanations, you’re not thinking critically enough.
• Verbal Heuristics: Words to help you think critically and/or dig up hidden assumptions.
• Mary Had a Little Lamb Heuristic – emphasize each word in that phrase and see where it takes you.
• Change “the” to “a” Heuristic:
  • “the killer bug” vs. “a killer bug”
  • “the deadline” vs. “a deadline”
• “Unless” Heuristic:  I’m done testing unless…you have other ideas
• “Except” Heuristic:  Every test must have expected results except those we have no idea what to expect from.
• “So Far” Heuristic:  I’m not aware of any problems…so far. 
• “Yet” Heuristic: Repeatable tests are fundamentally more valuable, yet they never seem to find bugs.
• “Compared to what?” Heuristic: Repeatable tests are fundamentally more valuable…compared to what?
• A tester’s job is to preserve uncertainty when everyone around us is certain.
• “Safety Language” is a precise way of speaking which differentiates between observation and inference.  Safety Language is a strong trigger for critical thinking.
  • “You may be right” is a great way to end an argument.
  • “It seems to me” is a great way to begin an observation.
  • Instead of “you should do this” try “you may want to do this”.
  • Instead of “it works” try “it meets the requirements to some degree”
  • All the verbal heuristics above can help us speak precisely.

Rapid Software Testing (RST) is for situations where you have to test a product right now, under conditions of uncertainty, in a way that stands up to scrutiny.

I don’t want to walk you through the exercises, videos, and discussions Michael Bolton used in class because…well, it’s his class, and you should take it!  But I will share some bite-sized test wisdom I wrote in my notebook during class.

Be careful, most of these are heuristic…

• An assumption is the opposite of a test.  (I love that!)
• Our job is not to be “done” with something.  It’s to find out interesting things that others do not already know.
• A tester’s job is to see the complexity behind the seemingly simple and to see the simplicity behind the seemingly complex.
• “Test” is a verb.  Not a noun.  It’s not an artifact.  It’s something one does.
• Testers do not put the quality in, they help others put the quality in.  Testers do not assure quality, but if we must use the term “QA”, maybe it should stand for Quality Assistance.
• Testers are like editors for writers.  No matter how well a writer tests their work, a good editor can normally find mistakes.
• Programmers do lots of testing.  But they need help finding the last few problems.
• A tester’s job is to remain uncertain, to maintain the idea that it might not work.
• providing a “QA Certification” is like your manager making you say “I will take the blame…”
• Testers don’t matter because the program is not intended for them.
• Discussions about quality are always political or emotional. – Jerry Weinberg
• An “Issue” is anything that threatens the value of our testing.  Issues should be reported.  They may be more important than bugs because they give bugs time to hide.
• Threats to testability are “issues”.  Two things that threaten testability are:
• Visibility (e.g., log files)
• Controllability – the capacity to make the program do stuff (e.g., can I update the DB and config files?)
• “Positive Test” – Fulfills every required assumption.  Entering a bad password is a positive test if we’ve already established how the bad password should be handled.
• What is testing?  Getting answers.
• A useful test approach:
• Know your mission
• Consider building a model of the product first
• Begin sympathetically
• Then chase risks
• The first few moments of a product should be based on learning.
• There’s always more than meets the eye.
• Maps and models that you build don’t have to be right.  They just need to get people thinking.
• If you don’t have enough time to test, one trick to get more time is to find important bugs.  People will generally delay releases.  (But don’t sit on bugs until the last minute, of course.  Report them as soon as you’re aware.)
• Don’t forget to “imploy the pause”.  Take the time to learn something new every now and then.

When someone walks up to your desk and asks, “How’s the testing going?”, a good answer depends on remembering to tell that person the right things.

After reading Michael Kelly’s post on coming up with a heuristic (model), I felt inspired to tweak his MCOASTER test reporting heuristic model.  I tried using it but it felt awkward.  I wanted one that was easier for me to remember, with slightly different trigger words, ordered better for my context.  It was fun.  Here is what I did:

1. I listed all the words that described things I may want to cover in a test report.  Some words were the same as Michael Kelly’s but some were different (e.g., “bugs”).
2. Then I took the first letter of each of my words and plugged them into Wordsmith.org’s anagram solver.
3. Finally, I skimmed through the anagram solver results, until I got to those starting with ‘M’ (I knew I wanted ‘Mission’ to be the first trigger).  If my mnemonic didn’t jump out at me, I tweaked the input words slightly using synonyms and repeated step 2.

Two minutes later I settled on MORE BATS.  After being an avid caver for some 13 years, it was perfect.  When someone asks for a test report, I feel slightly lost at first, until I see MORE BATS.  Just like in caving; when deep into the cave I feel lost, but seeing “more bats” is a good sign the entrance is in that general direction, because more bats tend to be near the entrance.

Here’s how it works (as a test report heuristic):

Mission

Obstacles

Risks

Environment

Bugs

Audience

Techniques

Status

“My mission is to test the new calendar control.  My only obstacle is finding enough data in the future for some of my scenarios.  The risks I’m testing include long range future entries, multiple date ranges, unknown begin dates, and extremely old dates.  I’m using the Dev environment because the control is not working in QA.  I found two date range bugs.  Since this is a programmer (audience) asking me for a test report I will describe my techniques; I’m checking my results in database tableA because the UI is not completed.  I think I’ll be done today (status).”

Try to make your own and let me know what you come up with.

CAST2011 was full of tester heavy weights.  Each time I sat down in the main gathering area, I picked a table with people I didn’t know.  One of those times I happened to sit down next to thetesteye.com blogger Henrik Emilsson.  After enjoying his conversation, I attended his Crafting Our Own Models of Software Quality track session. 

CRUSSPIC STMPL (pronounced Krusspic Stemple)…I had heard James Bach mention his quality criteria model mnemonic years ago.  CRUSSPIC represents operational quality criteria (i.e., Capability, Reliability, Usability, Security, Scalability, Performance, Installability, Compatibility).  STMPL represents development quality criteria (i.e., Supportability, Testability, Maintainability, Portability, Localizability). 

Despite how appealing it is to taste the phrase CRUSSPIC STMPL as it exercises the mouth, I had always considered it too abstract to benefit my testing. 

Henrik, on the other hand, did not agree.  He began his presentation quoting statistician George Edward Pelham Box, who said “…all models are wrong, but some are useful”.  Henrik believes we should all create models that are better for our context. 

With that, Henrik and his tester colleagues took Bach’s CRUSSPIC STMPL, and over the course of about a year, modified it to their own context.  Their current model, CRUCSPIC STMP, is posted here.  They spent countless hours reworking what each criterion means to them. 

They also swapped out some of the criteria for their own.  Of note, was swapping out the 4th “S” for a “C”; Charisma.  When you think about some of your favorite software products, charisma probably plays an important role.  Is it good-looking?  Do you get hooked and have fun?  Does the product have a compelling inception story (e.g., Facebook).  And to take CRUCSPIC STMP further, Henrik has worked in nested mnemonics.  The Charisma quality item descriptors are SPACE HEADS (i.e., Satisfaction, Professionalism, Attractiveness, Curiosity, Entrancement, Hype, Expectancy, Attitude, Directness, Story).

Impressive.  But how practical is it? 

After Henrik’s presentation, I have to admit, I’m convinced it has enough value for it’s efforts:

• Talking to customers - If quality is value to some person, a quality model can be used to help that person (customers/users) explain which quality criteria is most important to them.  This, in turn, will guide the tester.
• Test idea triggers - Per Henrik, a great model inspires you to think for yourself.
• Evaluating test results – If Concurrency is a target quality criterion, did my test tell me anything about performing parallel tasks?
• Talking about testing – Reputation and integrity are important traits for skilled testers.  When James Bach or Henrik Emilsson talk about testing, their intimate knowledge of their quality models gives them an air of sophistication that is hard to beat.

Yes, I’m inspired to build a quality criteria model.  Thank you, Henrik!

Sorry about the blog lapse. I just got back from STPCon and a 10-day vacation in the Pacific Northwest.

I’ll share my STPCon ideas and takeaways in future posts. But first I have to mention my STPCon highlight.

During the conference, keynote speaker James Bach invited me to participate in an experiment to see how quickly testers can learn testing techniques. He approached me before lunch and I joined him and another tester, Michael Richardson, in the hotel lobby. James asked us each to play a different game with him (simultaneously). Michael played the “Dice Game”, in which Michael had to roll random dice of Michael’s choosing, from several different dice styles, to determine a pattern that James made up.

I played a game much like “20 Questions”, in which I had to explain what happened based on vague information I was given. I could ask as many Yes/No questions as I wanted. During the games, James coached us on different techniques. After about 45 minutes, Michael jumped in and bailed me out. Afterwards, James asked me to play again with a different premise. I was able to crack the second premise in less than 5 minutes. I would like to think I actually learned something from my first, painful, attempt.

These games share similarities to software testing because they require skillful elimination of infinite variables. Some of the techniques James suggested were:

• Focus then defocus. Sometimes you should target a specific detail. Other times, you should step back and think less narrow. For me, defocus was the most important approach.
• Forward then backward. Search for evidence to backup a theory. Then search for a theory based on evidence you have determined.
• Dumb questions can lead to interesting answers. Don’t be afraid to ask as many questions as you can, even if they are seemingly stupid questions.
• Flat lining. If you are getting nowhere with a technique, it’s time to stop. Try a different type of testing.

Later, James asked Michael to think up a challenging dice game pattern. James was able to solve it in about 30 seconds using only one die. Obviously, having played before, he knew it would eliminate most of the variables. I just used this exact idea today to understand a software behavior problem I initially thought complex.

After the stress of performing these challenges with James, we sat back and enjoyed a conversation I will not forget. James was more personable and less judgemental than I expected. Despite being a high school dropout, he is currently writing a book about learning (not about testing). I also thought it cool that during STPCon he headed across the street to the Boston Public Library and began reading from various history books. He was trying to determine why people with power would ever choose to give it up. I guess he’s got more on his mind than just testing.

For me, spending time with James was a huge privilege and I was grateful for the opportunity to discuss testing with him, as well as getting to know the person behind the celebrity.

Note: Michael Richardson was also an interesting guy. He once did QA for a fish company and was literally smacked in the face with a large fish due to poor testing.

This tired question has several variations and here is what I think.

If the question is, “Am I done testing this AUT?” the answer is, of course, no. There are an infinite number of tests to execute, so there is no such thing as finishing early in testing. Sorry. You should still be executing tests as your manager takes away your computer and rips you from your cubicle in an effort to stop you from logging your next bug. Or as Ben Simo’s 12 year old daughter puts it, you’re not done testing until you die or get really tired of it.

The more realistic question, “Is it time to stop testing this AUT?” probably depends on today’s date. We do the best we can within the time we are given. Some outside constraint (e.g., project completion date, ship date, you are reassigned to a different role) probably provides your hard stop. The decision of when to stop testing is not left up to the tester, although your feedback was probably considered early on… when nothing was really known about the AUT.

Finally, the question, “Am I done testing this feature?” is much more interesting and valuable to the tester. Assuming your AUT has multiple features that are ready for testing, you’ll want to pace yourself so attention is given to all features, or at least all important features. This is a balancing game because too much time spent on any given feature may cause neglect on others. I like to use two heuristics to guide me.

Popcorn Heuristic – I heard this one at Michael Bolton’s excellent Rapid Software Testing class. How do we know when a bag of microwave popcorn is finished popping? Bugs are discovered in much the same way. We poke around and start finding a few bugs. We look a little deeper and suddenly bugs are popping up like crazy. Finally, they start getting harder to find. Stop the microwave. Move on to the next feature, we’re done here!

Straw House Heuristic – I picked up this one from Adam White’s blog. Don’t use a tornado to hit a straw house. When we first begin testing a new feature, we should poke at it with some simple sanity tests and see how it does. If it can’t stand up against the simple stuff, we may be wasting our time with further testing. It’s hard to resist the joy of flooding the bug tracking system with easy bugs but our skills would be wasted, right? Make sure your devs agree with your assessment that said feature is not quite ready for your tornado, log some high level bugs, and ask them if they’ll build you a stone house. Then move on to the next feature, we’re done here!

What do you think?