This efficiency didn’t occur to me until recently.  I was doing an exploratory test session and documenting my tests via Rapid Reporter.  My normal process had always been to document the test I was about to execute…

TEST: Edit element with unlinked parent

…execute the test.  Then write “PASS” or “FAIL” after it like this…

TEST: Edit element with unlinked parent – PASS

But it occurred to me that if a test appears to fail, I tag said failure as a “Bug”, “Issue”, “Question”, or “Next Time”.  As long as I do that consistently, there is no need to add “PASS” or “FAIL” to the documented tests.  While debriefing about my tests post session, the assumption will be that the test passed unless indicated otherwise.

Even though it felt like going to work without pants, after a few more sessions, it turns out, not resolving to “PASS” or “FAIL” reduces administrative time and causes no ambiguity during test reviews.  Cool!

Wait. It gets better.

On further analysis, resolving all my tests to “PASS” or “FAIL” may have prevented me from actual testing.  It was influencing me to frame everything as a check.  Real testing does not have to result in “PASS” or “FAIL”.  If I didn’t know what was supposed to happen after editing an element with an unlinked parent (as in the above example), well then it didn’t really “PASS” or “FAIL”, right?  However, I may have learned something important nevertheless, which made the test worth doing…I’m rambling.

The bottom line is, maybe you don’t need to indicate “PASS” or “FAIL”.  Try it.

(and one of Michael Bolton’s)
One of my testers took James Bach’s 3-day online Rapid Testing Intensive class.  I poked my head in from time to time, couldn’t help it.  What struck me is how metaphor after metaphor dripped from Bach’s mouth like poetry.  I’ve heard him speak more times than I can count but I’ve never heard such a spontaneous panoply of beautiful metaphors.  Michael Bolton, acting as assistant, chimed in periodically with his own metaphors.  Here are some from the portions I observed:

• A tester is like a smoke alarm, their job is to tell people when and where a fire is.  However, they are not responsible for telling people how to evacuate the building or put out the fire.
• (Michael Bolton) Testers are like scientists.  But scientists have it easy;  They only get one build.  Testers get new builds daily so all bets are off on yesterday’s test results.
• Buying test tools is like buying a sweater for someone.  The problem is, they feel obligated to wear the sweater, even if it’s not a good fit.
• Testers need to make a choice; either learn to write code or learn to be charming.  If you’re charming, perhaps you can get a programmer to write code for you.   It’s like having a friend who owns a boat.
• Deep vs. Shallow testing.  Some testers only do Shallow testing.  That is like driving a car with a rattle in the door…”I hear a rattle in the door but it seems to stay shut when I drive so…who cares?”.
• Asking a tester how long it will take to test is like being diagnosed with cancer and asking the doctor how long you have to live.
• Asking a tester how long the testing phase will last is like asking a flight attendant how long the flight attendant service will last.
• Complaining to the programmers about how bad their code looks is like being a patron at a restaurant and walking back into the kitchen to complain about the food to the chefs.  How do you think they’re going to take it?
• Too many testers and test managers want to rush to formality (e.g., test scripts, test plans).  It’s like wanting to teleport yourself home from the gym.  Take the stairs!

Well, that depends on what your clients need.  How do you know what your clients need?  I can think of two ways:

1. Ask them.  Be careful. Per my experience, clients inflate their test documentation needs when asked.  Maybe they’re afraid they’ll insult the tester if they don’t ask for lots of test documentation.  Maybe they’re intent to review test cases is stronger than their follow-through.
2. Observe them.  Do they ever ask for test case reviews?  If you are transparent with your test documentation, do your clients ever give you feedback?  Mine don’t (unless I initiate it).

Here is what I ask of my testers:

Document enough detail such that you can explain the testing you did, in-person, up to a year later.

Before I came up with the above, I started with this:  The tester should be present to explain their testing.  Otherwise, we risk incorrect information. 

If the tester will be present, why would we sacrifice test time to write details that can easily be explained by the tester?  In my case, the documentation serves to remind the tester.   When we review it with programmers, BA’s, users, other testers, auditors, or if I review it myself, the tester should always be present to interpret.

What if the tester leaves?

I’ll talk about that in the next post.

After RST class (see my Four Day With Michael Bolton post), Bolton did a short critical thinking for testers workshop.  If you get an opportunity to attend one of these at a conference or other place, it’s time well spent.  The exercises were great, but I won’t blog about them because I don’t want to give them away.  Here is what I found in my notes…

• There are two types of thinking:
  1. System 1 Thinking – You use it all the time to make quick answers.  It works fine as long as things are not complex.
  2. System 2 Thinking – This thinking is lazy, you have to wake it up.
• If you want to be excellent at testing, you need to use System 2 Thinking.  Testing is not a straight forward technical problem because we are creating stuff that is largely invisible.
• Don’t plan or execute tests until you obtain context about the test mission.
• Leaping to assumptions carries risk.  Don’t build a network of assumptions.
• Avoid assumptions when:
  • critical things depend on it
  • when the assumption is unlikely to be true
  • the assumption is dangerous when not declared
• Huh?  Really?  So?   (James Bach’s critical thinking heuristic)
  • Huh? – Do I really understand?
  • Really? – How do I know what you say is true?
  • So? – Is that the only solution?
• “Rule of Three” – If you haven't thought of at least three plausible explanations, you’re not thinking critically enough.
• Verbal Heuristics: Words to help you think critically and/or dig up hidden assumptions.
• Mary Had a Little Lamb Heuristic – emphasize each word in that phrase and see where it takes you.
• Change “the” to “a” Heuristic:
  • “the killer bug” vs. “a killer bug”
  • “the deadline” vs. “a deadline”
• “Unless” Heuristic:  I’m done testing unless…you have other ideas
• “Except” Heuristic:  Every test must have expected results except those we have no idea what to expect from.
• “So Far” Heuristic:  I’m not aware of any problems…so far. 
• “Yet” Heuristic: Repeatable tests are fundamentally more valuable, yet they never seem to find bugs.
• “Compared to what?” Heuristic: Repeatable tests are fundamentally more valuable…compared to what?
• A tester’s job is to preserve uncertainty when everyone around us is certain.
• “Safety Language” is a precise way of speaking which differentiates between observation and inference.  Safety Language is a strong trigger for critical thinking.
  • “You may be right” is a great way to end an argument.
  • “It seems to me” is a great way to begin an observation.
  • Instead of “you should do this” try “you may want to do this”.
  • Instead of “it works” try “it meets the requirements to some degree”
  • All the verbal heuristics above can help us speak precisely.

See Part 1 for intro.

• People don’t make decisions based on numbers, they do so based on feelings (about numbers).
• Asking for ROI numbers for test automation or social media infrastructure does not make sense because those are not investments, those are expenses.  Value from an automation tool is not quantifiable.  It does not replace a test a human can perform.  It is not even a test.  It is a “check”.
• Many people say they want a “metric” when what they really want is a “measurement”.  A “metric” allows you to stick a number on an observation.  A “measurement”, per Jerry Weinberg, is anything that allows us to make observations we can rely on.  A measurement is about evaluating the difference between what we have and what we think we have.
• If someone asks for a metric, you may want to ask them what type of information they want to know (instead of providing them with a metric).
• When something is presented as a “problem for testing”, try reframing it to “a problem testing can solve”.
• Requirements are not a thing.  Requirements are not the same as a requirements document.  Requirements are an abstract construct.  It is okay to say the requirements document is in conflict with the requirements.  Don’t ever say “the requirements are incomplete”.  Requirements are not something that can be incomplete.  Requirements are complete before you even know they exist, before anyone attempts to write a requirements document.
• Skilled testers can accelerate development by revealing requirements.  Who cares what the requirement document says.
• When testing, don’t get hung up on “completeness”.  Settle for adequate.  Same for requirement documents.  Example: Does your employee manual say “wear pants to work”?  Do you know how to get to your kid’s school without knowing the address?
• Session-Based Test Management (SBTM) emphasizes conversation over documentation.  It’s better to know where your kid’s school is than to know the address.
• SBTM requires 4 things:
• Charter
• Time-boxed test session
• Reviewable results
• Debrief
• The purpose of a program is to provide value to people.  Maybe testing is more than checking.
• Quality is more than the absence of bugs.
• Don’t tell testers to “make sure it works”.  Tell them to “find out where it won’t work.”  (yikes, that does rub against the grain with my We Test To Find Out If Software *Can* Work post, but I still believe each)
• Maybe when something goes wrong in production, it’s not the beginning of a crisis, it’s the end of an illusion.

See Part 1 for intro.

• There are two reasons why your bugs are not getting fixed:
1. There is more important stuff going on
2. You are not explaining them well.
• Testers need to be good at articulating why we think something is a bug.  One approach is PEW.  State the Problem, provide an Example, explain Why it matters.
• “How many test cases?” is usually a silly question.
• There are two reasons why all tests do not get executed: 
1. The tester didn’t think of it.
2. The tester thought of it but decided not to execute it.  Hopefully it’s the latter.  It may be worth while to brainstorm on tests.
• One way to communicate coverage to stakeholders is to use a mind map.
• If you get bored testing, you may be doing something wrong (e.g., you are doing repetitive tests, you are not finding anything interesting).
• Testing is about looking for a “problem”.  A “problem” is an undesirable situation that is solvable.
• (I need to stop being so militant about this) All bugs don’t need repro steps.  Repro steps may be expensive.
• Consider referencing your oracle (the way of recognizing a problem you used to find the bug) in your bug report.
• When asked to perform significantly time consuming or complex testing, consider the Orange Juice Test:  A client asked three different hotels if the hotels could supply said client with two thousand glasses of fresh squeezed orange juice tomorrow morning.  Hotel #1 said “no”.  Hotel #2 said “yes”.  Hotel #3 said “yes, but here’s what it’s going to cost you”.  The client didn’t really want orange juice.  They picked Hotel #3.
• No test can tell us about the future.
• Nobody really knows what 100% test coverage means.  Therefore, it may not make sense to describe test coverage as a percentage.  Instead, try explaining it as the extent to which we have travelled over some agreed upon map.  And don’t talk about coverage unless you talk about the kind of coverage you are talking about (e.g., Functions, Platforms, Data, Time, etc.)
• Asking how long a testing phase should be is like asking how long I have to look out the windshield as I drive to Seattle.
• Skilled testers are like crime scene investigators.  Testers are not in control (the police are).  Testers give the police the information they need.  If there is another crime committed, you may not have time to investigate as much with the current crime scene.
• No test can prove a theory is correct.  A test can only disprove it.
• (I still have a hard time with this one) Exploratory Testing (ET) is not an activity that one can do. It is not a technique.  It is an approach.  A test is exploratory if the ideas are coming from the tester in the here and now.  ET can be automated.  Scripts come from exploration.
• Exploratory behavior = Value seeking.
• Scripted behavior = Task seeking
• Tests should not be concerned with the repeatability of computers.  It’s important to induce variation.
• ET is a structured approach.  One of the most important structures is the testing story.  A skilled tester should be able to tell three stories:
1. A story about the product (e.g., is the product any good?).
2. A story about how you tested it (e.g., how do I know?  Because I tested it by doing this…).
3. A story about the value of the testing (e.g., here is why you should be pleased with my work…).

Rapid Software Testing (RST) is for situations where you have to test a product right now, under conditions of uncertainty, in a way that stands up to scrutiny.

I don’t want to walk you through the exercises, videos, and discussions Michael Bolton used in class because…well, it’s his class, and you should take it!  But I will share some bite-sized test wisdom I wrote in my notebook during class.

Be careful, most of these are heuristic…

• An assumption is the opposite of a test.  (I love that!)
• Our job is not to be “done” with something.  It’s to find out interesting things that others do not already know.
• A tester’s job is to see the complexity behind the seemingly simple and to see the simplicity behind the seemingly complex.
• “Test” is a verb.  Not a noun.  It’s not an artifact.  It’s something one does.
• Testers do not put the quality in, they help others put the quality in.  Testers do not assure quality, but if we must use the term “QA”, maybe it should stand for Quality Assistance.
• Testers are like editors for writers.  No matter how well a writer tests their work, a good editor can normally find mistakes.
• Programmers do lots of testing.  But they need help finding the last few problems.
• A tester’s job is to remain uncertain, to maintain the idea that it might not work.
• providing a “QA Certification” is like your manager making you say “I will take the blame…”
• Testers don’t matter because the program is not intended for them.
• Discussions about quality are always political or emotional. – Jerry Weinberg
• An “Issue” is anything that threatens the value of our testing.  Issues should be reported.  They may be more important than bugs because they give bugs time to hide.
• Threats to testability are “issues”.  Two things that threaten testability are:
• Visibility (e.g., log files)
• Controllability – the capacity to make the program do stuff (e.g., can I update the DB and config files?)
• “Positive Test” – Fulfills every required assumption.  Entering a bad password is a positive test if we’ve already established how the bad password should be handled.
• What is testing?  Getting answers.
• A useful test approach:
• Know your mission
• Consider building a model of the product first
• Begin sympathetically
• Then chase risks
• The first few moments of a product should be based on learning.
• There’s always more than meets the eye.
• Maps and models that you build don’t have to be right.  They just need to get people thinking.
• If you don’t have enough time to test, one trick to get more time is to find important bugs.  People will generally delay releases.  (But don’t sit on bugs until the last minute, of course.  Report them as soon as you’re aware.)
• Don’t forget to “imploy the pause”.  Take the time to learn something new every now and then.