One of my tester colleagues and I had an engaging discussion the other day. 

If a test failure is not caused by a problem in the system-under-test, should the tester bother to say the test failed? 

My position is: No. 

If a test fails but there is no problem with system-under-test, it seems to me it’s a bad test.  Fix the test or ignore the results.  Explaining that a test failure is nothing to be concerned with, gives the project team a net gain of nothing.  (Note: If the failure has been published, my position changes; the failure should be explained).

The context of our discussion was the test automation space. I think test automaters, for some reason, feel compelled to announce automated check failures in one breath, and in the next, explain why these failures should not matter.  “Two automated checks failed…but it’s because the data was not as expected, so I’m not concerned” or “ten automated checks are still failing but it’s because something in the system-under-test changed and the automated checks broke…so I’m not concerned”. 

My guess is, project teams and stakeholders don’t care if tests passed or failed.  They care about what those passes and failures reveal about the system-under-test.  See the difference?

Did the investigation of the failed test reveal anything interesting about the system-under-test?  If so, share what it revealed.  The fact that the investigation was triggered by a bad test is not interesting.

If we’re not careful, Test Automation can warp our behavior. IMO, a good way of understanding how to behave in the test automation space, is to pretend your automated checks are sapient (AKA “manual”) tests.  If a sapient tester gets different results than they expected, but later realizes their expectations were wrong, they don’t bother to explain their recent revelation to the project team.  A sapient tester would not say, “I thought I found a problem, but then I realized I didn’t”?  Does that help anyone?

My system 1 thinking says “no”.  I’ve often heard separation of duties makes testers valuable.

Let’s explore this.

A programmer and a tester are both working on a feature requiring a complex data pull.  The tester knows SQL and the business data better than the programmer.

If Testers Write Source Code:

The tester writes the query and hands it to the programmer.  Two weeks later, as part of the “testing phase”, the tester tests the query (they wrote themselves) and finds 0 bugs.  Is anything dysfunctional about that? 

If Testers do NOT Write Source Code:

The programmer struggles but manages to cobble some SQL together.  In parallel, the tester writes their own SQL and puts it in an automated check.  During the “testing phase”, the tester compares the results of their SQL with that of the programmer’s and finds 10 bugs.  Is anything dysfunctional about that? 

After RST class (see my Four Day With Michael Bolton post), Bolton did a short critical thinking for testers workshop.  If you get an opportunity to attend one of these at a conference or other place, it’s time well spent.  The exercises were great, but I won’t blog about them because I don’t want to give them away.  Here is what I found in my notes…

• There are two types of thinking:
  1. System 1 Thinking – You use it all the time to make quick answers.  It works fine as long as things are not complex.
  2. System 2 Thinking – This thinking is lazy, you have to wake it up.
• If you want to be excellent at testing, you need to use System 2 Thinking.  Testing is not a straight forward technical problem because we are creating stuff that is largely invisible.
• Don’t plan or execute tests until you obtain context about the test mission.
• Leaping to assumptions carries risk.  Don’t build a network of assumptions.
• Avoid assumptions when:
  • critical things depend on it
  • when the assumption is unlikely to be true
  • the assumption is dangerous when not declared
• Huh?  Really?  So?   (James Bach’s critical thinking heuristic)
  • Huh? – Do I really understand?
  • Really? – How do I know what you say is true?
  • So? – Is that the only solution?
• “Rule of Three” – If you haven't thought of at least three plausible explanations, you’re not thinking critically enough.
• Verbal Heuristics: Words to help you think critically and/or dig up hidden assumptions.
• Mary Had a Little Lamb Heuristic – emphasize each word in that phrase and see where it takes you.
• Change “the” to “a” Heuristic:
  • “the killer bug” vs. “a killer bug”
  • “the deadline” vs. “a deadline”
• “Unless” Heuristic:  I’m done testing unless…you have other ideas
• “Except” Heuristic:  Every test must have expected results except those we have no idea what to expect from.
• “So Far” Heuristic:  I’m not aware of any problems…so far. 
• “Yet” Heuristic: Repeatable tests are fundamentally more valuable, yet they never seem to find bugs.
• “Compared to what?” Heuristic: Repeatable tests are fundamentally more valuable…compared to what?
• A tester’s job is to preserve uncertainty when everyone around us is certain.
• “Safety Language” is a precise way of speaking which differentiates between observation and inference.  Safety Language is a strong trigger for critical thinking.
  • “You may be right” is a great way to end an argument.
  • “It seems to me” is a great way to begin an observation.
  • Instead of “you should do this” try “you may want to do this”.
  • Instead of “it works” try “it meets the requirements to some degree”
  • All the verbal heuristics above can help us speak precisely.

See Part 1 for intro.

• People don’t make decisions based on numbers, they do so based on feelings (about numbers).
• Asking for ROI numbers for test automation or social media infrastructure does not make sense because those are not investments, those are expenses.  Value from an automation tool is not quantifiable.  It does not replace a test a human can perform.  It is not even a test.  It is a “check”.
• Many people say they want a “metric” when what they really want is a “measurement”.  A “metric” allows you to stick a number on an observation.  A “measurement”, per Jerry Weinberg, is anything that allows us to make observations we can rely on.  A measurement is about evaluating the difference between what we have and what we think we have.
• If someone asks for a metric, you may want to ask them what type of information they want to know (instead of providing them with a metric).
• When something is presented as a “problem for testing”, try reframing it to “a problem testing can solve”.
• Requirements are not a thing.  Requirements are not the same as a requirements document.  Requirements are an abstract construct.  It is okay to say the requirements document is in conflict with the requirements.  Don’t ever say “the requirements are incomplete”.  Requirements are not something that can be incomplete.  Requirements are complete before you even know they exist, before anyone attempts to write a requirements document.
• Skilled testers can accelerate development by revealing requirements.  Who cares what the requirement document says.
• When testing, don’t get hung up on “completeness”.  Settle for adequate.  Same for requirement documents.  Example: Does your employee manual say “wear pants to work”?  Do you know how to get to your kid’s school without knowing the address?
• Session-Based Test Management (SBTM) emphasizes conversation over documentation.  It’s better to know where your kid’s school is than to know the address.
• SBTM requires 4 things:
• Charter
• Time-boxed test session
• Reviewable results
• Debrief
• The purpose of a program is to provide value to people.  Maybe testing is more than checking.
• Quality is more than the absence of bugs.
• Don’t tell testers to “make sure it works”.  Tell them to “find out where it won’t work.”  (yikes, that does rub against the grain with my We Test To Find Out If Software *Can* Work post, but I still believe each)
• Maybe when something goes wrong in production, it’s not the beginning of a crisis, it’s the end of an illusion.

After failing in production for a third time, the team lead’s passive aggressive tendencies became apparent in his bug report title.  Can you blame him?

It all depends on context, of course.  But if three attempts to get something working in production still fail…there may be a larger problem somewhere. 

That got me thinking.  Maybe we should add passive aggressive suffixes for all our  “escapes” (bugs not caught in test).  It would serve to embarrass and remind ourselves that we can do better.

• “…fail I” would not be so bad.
• “…fail II” would be embarrassing.
• “…fail III” should make us ask for help testing and coding.
• “…fail IV” should make us ask to be transferred to a more suitable project.
• by “…fail V” we should be taking our users out to lunch.
• “…fail VI” I’ve always wanted to be a marine biologist, no time like the present.

See Part 1 for intro.

• There are two reasons why your bugs are not getting fixed:
1. There is more important stuff going on
2. You are not explaining them well.
• Testers need to be good at articulating why we think something is a bug.  One approach is PEW.  State the Problem, provide an Example, explain Why it matters.
• “How many test cases?” is usually a silly question.
• There are two reasons why all tests do not get executed: 
1. The tester didn’t think of it.
2. The tester thought of it but decided not to execute it.  Hopefully it’s the latter.  It may be worth while to brainstorm on tests.
• One way to communicate coverage to stakeholders is to use a mind map.
• If you get bored testing, you may be doing something wrong (e.g., you are doing repetitive tests, you are not finding anything interesting).
• Testing is about looking for a “problem”.  A “problem” is an undesirable situation that is solvable.
• (I need to stop being so militant about this) All bugs don’t need repro steps.  Repro steps may be expensive.
• Consider referencing your oracle (the way of recognizing a problem you used to find the bug) in your bug report.
• When asked to perform significantly time consuming or complex testing, consider the Orange Juice Test:  A client asked three different hotels if the hotels could supply said client with two thousand glasses of fresh squeezed orange juice tomorrow morning.  Hotel #1 said “no”.  Hotel #2 said “yes”.  Hotel #3 said “yes, but here’s what it’s going to cost you”.  The client didn’t really want orange juice.  They picked Hotel #3.
• No test can tell us about the future.
• Nobody really knows what 100% test coverage means.  Therefore, it may not make sense to describe test coverage as a percentage.  Instead, try explaining it as the extent to which we have travelled over some agreed upon map.  And don’t talk about coverage unless you talk about the kind of coverage you are talking about (e.g., Functions, Platforms, Data, Time, etc.)
• Asking how long a testing phase should be is like asking how long I have to look out the windshield as I drive to Seattle.
• Skilled testers are like crime scene investigators.  Testers are not in control (the police are).  Testers give the police the information they need.  If there is another crime committed, you may not have time to investigate as much with the current crime scene.
• No test can prove a theory is correct.  A test can only disprove it.
• (I still have a hard time with this one) Exploratory Testing (ET) is not an activity that one can do. It is not a technique.  It is an approach.  A test is exploratory if the ideas are coming from the tester in the here and now.  ET can be automated.  Scripts come from exploration.
• Exploratory behavior = Value seeking.
• Scripted behavior = Task seeking
• Tests should not be concerned with the repeatability of computers.  It’s important to induce variation.
• ET is a structured approach.  One of the most important structures is the testing story.  A skilled tester should be able to tell three stories:
1. A story about the product (e.g., is the product any good?).
2. A story about how you tested it (e.g., how do I know?  Because I tested it by doing this…).
3. A story about the value of the testing (e.g., here is why you should be pleased with my work…).

Rapid Software Testing (RST) is for situations where you have to test a product right now, under conditions of uncertainty, in a way that stands up to scrutiny.

I don’t want to walk you through the exercises, videos, and discussions Michael Bolton used in class because…well, it’s his class, and you should take it!  But I will share some bite-sized test wisdom I wrote in my notebook during class.

Be careful, most of these are heuristic…

• An assumption is the opposite of a test.  (I love that!)
• Our job is not to be “done” with something.  It’s to find out interesting things that others do not already know.
• A tester’s job is to see the complexity behind the seemingly simple and to see the simplicity behind the seemingly complex.
• “Test” is a verb.  Not a noun.  It’s not an artifact.  It’s something one does.
• Testers do not put the quality in, they help others put the quality in.  Testers do not assure quality, but if we must use the term “QA”, maybe it should stand for Quality Assistance.
• Testers are like editors for writers.  No matter how well a writer tests their work, a good editor can normally find mistakes.
• Programmers do lots of testing.  But they need help finding the last few problems.
• A tester’s job is to remain uncertain, to maintain the idea that it might not work.
• providing a “QA Certification” is like your manager making you say “I will take the blame…”
• Testers don’t matter because the program is not intended for them.
• Discussions about quality are always political or emotional. – Jerry Weinberg
• An “Issue” is anything that threatens the value of our testing.  Issues should be reported.  They may be more important than bugs because they give bugs time to hide.
• Threats to testability are “issues”.  Two things that threaten testability are:
• Visibility (e.g., log files)
• Controllability – the capacity to make the program do stuff (e.g., can I update the DB and config files?)
• “Positive Test” – Fulfills every required assumption.  Entering a bad password is a positive test if we’ve already established how the bad password should be handled.
• What is testing?  Getting answers.
• A useful test approach:
• Know your mission
• Consider building a model of the product first
• Begin sympathetically
• Then chase risks
• The first few moments of a product should be based on learning.
• There’s always more than meets the eye.
• Maps and models that you build don’t have to be right.  They just need to get people thinking.
• If you don’t have enough time to test, one trick to get more time is to find important bugs.  People will generally delay releases.  (But don’t sit on bugs until the last minute, of course.  Report them as soon as you’re aware.)
• Don’t forget to “imploy the pause”.  Take the time to learn something new every now and then.