Last week, Alex Kell (Atlanta-based tester and my former boss) gave a fun talk at Software Testing Club Atlanta, ”The Oracle is Fallible: Recognizing, Understanding, and Evaluating the Assumptions that Testers Make”.

File:John William Waterhouse oracle 1884.png

Here are the highlights from my notes:

  • After showing John William Waterhouse’s famous 1884 painting, Consulting the Oracle (above), of 7 priestesses listening to an 8th priestess (playing the Oracle) interpret from the gods, Alex asked:
  • “Assumptions, are they bad or good?”
    • We make them because we’re lazy.
    • Sometimes we know we’re making an assumption, sometimes we don’t know.
    • After some discussion and examples of assumptions we realized we are all constantly making assumptions every waking moment and decided assumptions can be good or bad.
  • Bad assumptions (or forgetting the Oracle is fallible):
    • “The spec is correct.” – Be careful.  Remember Ron Jeffries “Three Cs”:
      • The Spec (AKA “Card”) is a reminder to have a conversation about something.
      • The Conversation is a discussion of the details that result in test confirmations.
      • The Confirmation is acceptance criteria that can be turned into acceptance tests.
    • “They know what they’re doing.” – What if everybody on the team is thinking this…group think?
    • “I know what I’m doing.”
    • “The software is working because we haven't seen a (red) failed test.”  (Dennis Stevens says “Every project starts red, and it stays red, until it is green.”)
    • “The model is reality.” – A model is an abstraction.  All decisions based on models are based on assumptions.  You should never be surprised when a model does not turn out to reflect reality.  Author, Nassim Nicholas Taleb, coined the word “platonicity” to describe the human tendency to find patterns in everything.

      Alex gave a nearly literal example of where people fall victim to this bad assumption: He told of people on Craig’s List (or similar) paying money for things like actual cars one can drive, only to discover they had just purchased a scaled down model of a car.
  • Good assumptions (I loved this and thought it was pretty bold to declare some assumptions being good for testers):
    • “The estimates are accurate”. – Take what you did last time.  Use the estimate until it is no longer helpful.
    • “The web service will honor its contract”. If testers didn’t make this assumption, might they be wasting time testing the wrong thing? 
    • There were more good assumptions but I have a gap in my notes.  Maybe Alex will leave a comment with those I missed.
  • Alex talked about J.B. Rainsberger’s “Integrated Tests Are a Scam” – In other words, if we don’t make some assumptions, we would have to code tests for the rest of our lives to make a dent in our coverage.
  • Suggestions to deal with assumptions:
    • Be explicit about your assumptions.
    • Use truth tables for complex scenarios (Alex shared one he used for his own testing).
    • System thinking – Testers should be able to explain the whole system.  This cuts down on bad assumptions.

1 comments:

  1. Carl said...

    Hopefully this will not post twice. Great post! I have not seen a truth table in the context of testing. I could not readily find an example via Google. Could you or Alex share an example?



Copyright 2006| Blogger Templates by GeckoandFly modified and converted to Blogger Beta by Blogcrowds.
No part of the content or the blog may be reproduced without prior written permission.