Last week, Alex Kell (Atlanta-based tester and my former boss) gave a fun talk at Software Testing Club Atlanta, ”The Oracle is Fallible: Recognizing, Understanding, and Evaluating the Assumptions that Testers Make”.
Here are the highlights from my notes:
- After showing John William Waterhouse’s famous 1884 painting, Consulting the Oracle (above), of 7 priestesses listening to an 8th priestess (playing the Oracle) interpret from the gods, Alex asked:
- “Assumptions, are they bad or good?”
- We make them because we’re lazy.
- Sometimes we know we’re making an assumption, sometimes we don’t know.
- After some discussion and examples of assumptions we realized we are all constantly making assumptions every waking moment and decided assumptions can be good or bad.
- Bad assumptions (or forgetting the Oracle is fallible):
- “The spec is correct.” – Be careful. Remember Ron Jeffries “Three Cs”:
- The Spec (AKA “Card”) is a reminder to have a conversation about something.
- The Conversation is a discussion of the details that result in test confirmations.
- The Confirmation is acceptance criteria that can be turned into acceptance tests.
- “They know what they’re doing.” – What if everybody on the team is thinking this…group think?
- “I know what I’m doing.”
- “The software is working because we haven't seen a (red) failed test.” (Dennis Stevens says “Every project starts red, and it stays red, until it is green.”)
- “The model is reality.” – A model is an abstraction. All decisions based on models are based on assumptions. You should never be surprised when a model does not turn out to reflect reality. Author, Nassim Nicholas Taleb, coined the word “platonicity” to describe the human tendency to find patterns in everything.
Alex gave a nearly literal example of where people fall victim to this bad assumption: He told of people on Craig’s List (or similar) paying money for things like actual cars one can drive, only to discover they had just purchased a scaled down model of a car.
- Good assumptions (I loved this and thought it was pretty bold to declare some assumptions being good for testers):
- “The estimates are accurate”. – Take what you did last time. Use the estimate until it is no longer helpful.
- “The web service will honor its contract”. If testers didn’t make this assumption, might they be wasting time testing the wrong thing?
- There were more good assumptions but I have a gap in my notes. Maybe Alex will leave a comment with those I missed.
- Alex talked about J.B. Rainsberger’s “Integrated Tests Are a Scam” – In other words, if we don’t make some assumptions, we would have to code tests for the rest of our lives to make a dent in our coverage.
- Suggestions to deal with assumptions:
- Be explicit about your assumptions.
- Use truth tables for complex scenarios (Alex shared one he used for his own testing).
- System thinking – Testers should be able to explain the whole system. This cuts down on bad assumptions.