Last week, Alex Kell (Atlanta-based tester and my former boss) gave a fun talk at Software Testing Club Atlanta, ”The Oracle is Fallible: Recognizing, Understanding, and Evaluating the Assumptions that Testers Make”.

Here are the highlights from my notes:

• After showing John William Waterhouse’s famous 1884 painting, Consulting the Oracle (above), of 7 priestesses listening to an 8th priestess (playing the Oracle) interpret from the gods, Alex asked:
• “Assumptions, are they bad or good?”
• We make them because we’re lazy.
• Sometimes we know we’re making an assumption, sometimes we don’t know.
• After some discussion and examples of assumptions we realized we are all constantly making assumptions every waking moment and decided assumptions can be good or bad.
• Bad assumptions (or forgetting the Oracle is fallible):
• “The spec is correct.” – Be careful.  Remember Ron Jeffries “Three Cs”:
• The Spec (AKA “Card”) is a reminder to have a conversation about something.
• The Conversation is a discussion of the details that result in test confirmations.
• The Confirmation is acceptance criteria that can be turned into acceptance tests.
• “They know what they’re doing.” – What if everybody on the team is thinking this…group think?
• “I know what I’m doing.”
• “The software is working because we haven't seen a (red) failed test.”  (Dennis Stevens says “Every project starts red, and it stays red, until it is green.”)
• “The model is reality.” – A model is an abstraction.  All decisions based on models are based on assumptions.  You should never be surprised when a model does not turn out to reflect reality.  Author, Nassim Nicholas Taleb, coined the word “platonicity” to describe the human tendency to find patterns in everything.
  
  Alex gave a nearly literal example of where people fall victim to this bad assumption: He told of people on Craig’s List (or similar) paying money for things like actual cars one can drive, only to discover they had just purchased a scaled down model of a car.
• Good assumptions (I loved this and thought it was pretty bold to declare some assumptions being good for testers):
• “The estimates are accurate”. – Take what you did last time.  Use the estimate until it is no longer helpful.
• “The web service will honor its contract”. If testers didn’t make this assumption, might they be wasting time testing the wrong thing? 
• There were more good assumptions but I have a gap in my notes.  Maybe Alex will leave a comment with those I missed.
• Alex talked about J.B. Rainsberger’s “Integrated Tests Are a Scam” – In other words, if we don’t make some assumptions, we would have to code tests for the rest of our lives to make a dent in our coverage.
• Suggestions to deal with assumptions:
• Be explicit about your assumptions.
• Use truth tables for complex scenarios (Alex shared one he used for his own testing).
• System thinking – Testers should be able to explain the whole system.  This cuts down on bad assumptions.

Last week, at STARCanada, I met several enthusiastic testers who might make great testing conference speakers.  We need you.  Life is too short for crappy conference talks.

I’m no pro by any means.  But I have been a track speaker at STARWest,  STARCanada, STPCon, and will be speaking at STAREast in 2 weeks. 

Ready to give it a go?  Here is my advice on procuring your first speaking slot:

1. Get some public speaking experience.  They are probably not going to pick you without speaking experience.  If you need experience, try speaking to a group of testers at your own company, at an IT group that meets within your city, volunteer for an emerging topic talk or sign up for a lightning talk at a conference that offers those, like CAST.
2. Come up with a killer topic.  See what speakers are currently talking about and talk about something fresh.  Make sure your topic can appeal to a wider audience.  Experience reports seem appealing.
3. Referrals – meet some speakers or industry leaders with some clout and ask them to review your talk.  If they like it, maybe they would consider putting in a good word for you.
4. Pick one or more conferences and search for their speaker submission deadlines and forms (e.g., Speaking At SQE Conferences).  If you’ve attended conferences, you are probably already on their mailing list and may be receiving said requests.  I’m guessing the 2014 SQE conference speaker submission will open in a few months.
5. Submit the speaker submission form.  Make sure you have an interesting sounding title.  You’ll be asked for a summary of your talk including take-aways and maybe how you intend to give it.  This is a good place to offer something creative about the way you will deliver your topic (e.g., you made a short video, you will do a hands-on group exercise).
6. Wait.  Eventually you’ll receive a call or email.  Sound competent.  Know your topic and be prepared to answer tough questions about it.
7. If you get rejected.  Politely ask what you could do differently to have a better chance of getting picked in the future.

It is not easy to get picked.  I was rejected several times and eventually got a nice referral from Lynn McKee, an experienced speaker with a great reputation; that helped.  One of my friends and colleagues, who is far more capable than I am, IMO, has yet to get picked up as a speaker.  So I don’t know what secret sauce they are looking for.

Good luck!

BTW - Speaking at conferences has both advantages and disadvantages to consider.

Advantages:

• The opportunity to build your reputation as an expert of sorts in the testing community.
• It helps you refine your ideas and possibly spread knowledge.
• Free registration fees.  This makes it more likely your company will pay your hotel/travel costs and let you attend.

Disadvantages:

• Public speaking is scary as hell for most of us.  The weeks leading up to a conference can be stressful.
• Putting together good talks and practicing takes lots of time.  I took days off work to prepare.

For most of us, testing for “coolness” is not at the top of our quality list.  Our users don’t have to buy what we test.  Instead, they get forced to use it by their employer.  Nevertheless, coolness can’t hurt. 

As far as testing for it…good luck.  It does not appear to be as straightforward as some may think.

I attended a mini-UX Conference earlier this week and saw Karen Holtzblatt, CEO and founder of InContext, speak.  Her keynote was the highlight of the conference for me, mostly because she was fun to watch.  She described the findings of 90 interviews and 2000 survey results, where her company asked people to show them “cool” things and explain why they considered them cool.

Her conclusion was that software aesthetics are way less important than the following four aspects:

1. Accomplishments – When using your software, people need to feel a sense of accomplishment without disrupting the momentum of their lives.  They need to feel like they are getting something done that was otherwise difficult.  They need to do this without giving up any part of their life.  Example: Can they accomplish something while waiting in line?
2. Connection – When using your software, they should be motivated to connect with people they actually care about (e.g., not Facebook friends).  These connections should be enriched in some manner.  Example: Were they able to share it with Mom?  Did they talk about it over Thanksgiving dinner?
3. Identity - When using your software, they should feel like they’re not alone.  They should be asking themselves, “Who am I?”, “Do I fit in with these other people?”.  They should be able to share their identity with joy.
4. Sensation – When using your software, they should experience a core sensory pleasure.  Examples: Can they interact with it in a fresh way via some new interface?  Can they see or hear something delightful?

Here are a few other notes I took:

• Modern users have no tolerance for anything but the most amazing experience.
• The app should help them get from thought to action, nothing in between.
• Users expect software to gather all the data they need and think for them.

I guess maybe I’ll think twice the next time I feel like saying, “just publish the user procedures, they’ll get it eventually”.

See Part 1 for intro.

• People don’t make decisions based on numbers, they do so based on feelings (about numbers).
• Asking for ROI numbers for test automation or social media infrastructure does not make sense because those are not investments, those are expenses.  Value from an automation tool is not quantifiable.  It does not replace a test a human can perform.  It is not even a test.  It is a “check”.
• Many people say they want a “metric” when what they really want is a “measurement”.  A “metric” allows you to stick a number on an observation.  A “measurement”, per Jerry Weinberg, is anything that allows us to make observations we can rely on.  A measurement is about evaluating the difference between what we have and what we think we have.
• If someone asks for a metric, you may want to ask them what type of information they want to know (instead of providing them with a metric).
• When something is presented as a “problem for testing”, try reframing it to “a problem testing can solve”.
• Requirements are not a thing.  Requirements are not the same as a requirements document.  Requirements are an abstract construct.  It is okay to say the requirements document is in conflict with the requirements.  Don’t ever say “the requirements are incomplete”.  Requirements are not something that can be incomplete.  Requirements are complete before you even know they exist, before anyone attempts to write a requirements document.
• Skilled testers can accelerate development by revealing requirements.  Who cares what the requirement document says.
• When testing, don’t get hung up on “completeness”.  Settle for adequate.  Same for requirement documents.  Example: Does your employee manual say “wear pants to work”?  Do you know how to get to your kid’s school without knowing the address?
• Session-Based Test Management (SBTM) emphasizes conversation over documentation.  It’s better to know where your kid’s school is than to know the address.
• SBTM requires 4 things:
• Charter
• Time-boxed test session
• Reviewable results
• Debrief
• The purpose of a program is to provide value to people.  Maybe testing is more than checking.
• Quality is more than the absence of bugs.
• Don’t tell testers to “make sure it works”.  Tell them to “find out where it won’t work.”  (yikes, that does rub against the grain with my We Test To Find Out If Software *Can* Work post, but I still believe each)
• Maybe when something goes wrong in production, it’s not the beginning of a crisis, it’s the end of an illusion.

I finally pulled it off!  My company brought Michael Bolton to teach a private 3-day Rapid Software Testing course and stick around for a 4th day of workshops and consulting.  On the fourth day I had Michael meet with QA Managers to give his talk/discussion on “How to Get The Best Value From Testing”.  Then he gave a talk for programmers, BAs, testers, and managers on “The Metrics Minefield”.  Finally, he did a 2.5 hour workshop on “Critical Thinking for Testers”.

My brain and pen were going the whole four days; every other sentence he uttered held some bit of testing wisdom.  I’ll post chunks of it in the near future.  I attended the class 6 years earlier in Toronto and I was concerned it would have the same material but fortunately most of it had changed.

The conversations before/after class were a real treat too.  After the first day, Claire Moss, Alex Kell, Michael Bolton, and I met at Fado for some Guinness, tester craic, and much to my surprise, to listen to Michael play mandolin in an Irish tradition music session.  He happened to be a very good musician and (of course) gave us handles to tell a slip jig from a reel.

Several days later, I’m still haunted by Michael-Bolton-speak.  I keep starting all my sentences with “it seems to me”.  But best of all perhaps, is the lingering inspiration to read, challenge, and contribute thoughtful ideas to our testing craft.  He got me charged up enough to love testing for at least another 6 years.  Thanks, Michael!

Jon Bach walked up to the podium and (referring to his readiness as the presenter) asked us how to tell the difference between a tester and a programmer:  A programmer would say, “I’m not ready for you guys yet”.

STPCon Spring 2012 kicked off with the best keynote I have seen yet.  Jon took on the recent Test-Is-Dead movement using a Journalism-Is-Dead metaphor.

He opened with the observation, “Did anyone get a ‘USA Today’ delivered to their room this morning?”

“No”. (something as a tester I was embarrassed not to have noticed.)

And after a safety language exercise, Jon presented a fresh testing definition, which reflects his previous career, journalism:

Testing is an interrogation and investigation in pursuit of information to aid evaluation.

Jon wondered out loud what had motivated the Test-Is-Dead folks.  “Maybe there is a lot of bad testing in our midst.”  And he proceeded to examine about 7 threats (I think there were more) that he believed could actually make testing dead.  Each testing threat was reinforced with its metaphorical journalism threat and coupled with a quote from the Test-Is-Dead folks. 

(I listed each threat-to-testing in bold text, followed by its journalism threat.  I listed an example Test-Is-Dead quote for threat #3 below.) 

1. (threat to testing) If the value of testing become irrelevant – (threat to journalism) If we stop caring about hearing the news of what is happening in the world. (implied: then testing and journalism is dead)
2. If the quality of testing is so poor that it suffers an irreversible “reputation collapse event”.  If “journalist” comes to mean “anybody who writes” (e.g., blogs, tweets, etc.).
3. If all users become early adaptors with excellent technical abilities.  If everyone becomes omnipotent; they already know today’s weather and tomorrow’s economic news.
   
   For this threat, the Test-Is-Dead quote was from James Whittaker, “You are a tester pretending to be a user”.  The context of Whittaker’s statement was that testers may not be as important because they are only pretending to be users, while modern technology may allow actual users to perform the testing.  Bach’s counterpoint was: since not all users may want to be testers and not all users may possess the skills to test, there may still be value for a tester role.
   
4. If testers are forced to channel all thoughts and intelligence through a limited set of tools and forced to only test what can be written as “executable specifications”.  If journalists could only report what the state allows.  Jon listed the example of the Egyptian news anchor that just resigned from state media after 20 years, due to what she called “lack of ethical standards” in the media’s coverage of the Arab Spring.
5. If all the tests testers could think of were confirmatory.  If all the journalists did not dig deeper (e.g., If they always assumed the story was just a car crash.)
6. If software stops changing and there is no need to ask new questions.  If the decisions people made today no longer depend on the state of the economy, weather, who they want to elect, etc.
7. If the craft of testing is made to be uninviting; into a boring clerical activity that smart, talented, motivated, creative people are not interested in.  If you had to file a “news release approval” form or go through the news czar for all the news stories you told.

Jon’s talk had some other highlights for me:

• He shared a list of tests he performed on eBay’s site prior to his eBay interview (e.g., can I find the most expensive item for sale?).  Apparently, he reported his test results during the interview.  This is an awesome idea.  If anyone did that to me, I would surely hire them.
• He also showed a list of keynote presentation requirements he received from STPCon.  He explained how these requirements (e.g., try to use humor in your presentation) were like tests.  Then he used the same metaphor to contrast those “tests” with “checks”; am I in the right room?  Is the microphone on?  Do I have a glass of water?

Jon concluded where he started.  He revealed that although newspapers may be dead, journalism is not.  Those journalists are just reporting the news differently.  And maybe it’s time to cut those unskilled testers loose as well.  But, according to Jon, the testing need for exploration and sapience in a rapid development world is more important than ever.

Hey conference haters.  Maybe it’s you…

I just got back from another awesome testing conference, Spring STPCon 2012 in New Orleans.  Apparently not all attendees shared my positive experience.  Between track sessions I heard the usual gripes:

“It’s not technical enough!”

“I expected the presenter to teach me how to install a tool and start writing tests with it.”

“It was just another Agile hippy love fest.”

“He just came up with fancy words to describe something I already do.”

I used to whine with the best of them.  Used to.  But now I have a blast and return full of ideas and inspiration.  Here are my suggestions on how to attend a testing conference and get the most out of it:

• Look for ideas, not instructions.  Adjust your expectations.  You are not going to learn how to script in Ruby.  That is something you can learn on your own.  Instead, you are going to learn how one tester used Ruby to write automated and manual API-layer REST service checks.
• Follow the presenters.  Long before the conference, select the track sessions you are interested in.  Find each presenter’s testing blog and/or Twitter name and follow them for several weeks.  Compare them and discard the duds.
• Talk to the presenters.  At the conference, use your test observation skills to identify presenters.  Introduce yourself and ask questions related to your project back at the office.  If you did my second bulleted suggestion above, you now have an ice-breaker, “Hey, I read your blog post about crowd source testing, I’m not sure I agree…”.
• Attend the non-track-session stuff too.  I think track sessions are the least interesting part of conferences.  The most interesting, entertaining, and easily digestible parts are the Lightning Talks, Speed Geeking, Breakfast Bytes, meal discussion tables, tester games, and keynotes.  Don’t miss these.
• Take notes.  Take waaaaaay more notes than you think you need.  I bring a little book and write non-stop during presentations.  It keeps me awake and engaged.  I can flip through said book on the plane, even when forced to turn off all personal electronics.
• Log Ideas.  Sometimes ideas are directly given during presentations.  But mostly, they come to you while applying information from presentations to your own situation.  I write the word “IDEA” in my book, followed by the idea.  Sometime these ideas have nothing to do with the presentation context.
• Don’t flee the scene.  When the conference ends each day, stick around.  You’ll generally find the big thinkers, still talking about testing in an informal hallway discussion.  I am uncomfortable in social situations and always feel awkward/intimidated by these folks but they are generally thrilled to bend your ear.
• Mix and mingle.  Again, I find parties and social situations extremely scary.  Despite that fear, I almost always make it a point to eat my conference meal with a group of people I’ve never seen before.  It always starts awkward but it ends with some new friends, business cards, and the realization that other testers are just as unsophisticated as I am.
• Submit a presentation.  If you hated one or more track sessions, channel that hate into your own presentation.  Take all the things you hated and do the opposite.  I did.  I got sick of always seeing consultants, vendors, and people who work for big fancy software companies.  So I pitched the opposite.  The real trick here is if you get accepted, the conference is free.  Let’s see your boss turn that one down.
• Play tester games or challenges.  If James Bach, Michael Bolton, or any of the other popular context-driven approach testers are attending the conference, tell them you are interested in playing tester games.  They are usually happy to coach you on testing skills in a fun way.  It may be a refreshing break from track sessions.
• Write a thank you card to your boss.  Don’t send an email.  Send something distinctive.  Let them know how much you appreciate their training money.  Tell them a few things you learned.  Tell them about my next bullet.
• Share something with your team.  The prospect of sharing your conference takeaways with your team will keep you motivated to learn during the conference and help you put those ideas to use.

What do you do to get the most out of your testing conference experiences?

My tech department had a little internal one day idea conference this week.  I wanted to keep it from being a programmer fest so I pitched Crowdsource Testing as a topic.  Although I knew nothing about it, I had just seen James Whittaker’s STARwest Keynote, and the notion of having non-testers test my product had been bouncing around my brain.

With a buzzword like “crowdsource” I guess I shouldn’t be surprised they picked it.  That and maybe I was one of their only submissions.

Due to several personal conflicts I had little time to research, but I did spend about 12 hours alone with my brain in a long car ride.  During that time I came up with a few ideas, or at least built on other people’s.

• Would you hang this on your wall?
  
  
  
  …only a handful of people raised their hands. But this is a Jackson Pollock!  Worth millions! It’s quality art!  Maybe software has something in common with art.
• Two testing challenges that can probably not be solved by skilled in-house testers are:
• Quality is subjective. (I tip my hat to Michael Bolton and Jerry Weinberg)
• There are an infinite amount of tests to execute. (I tip my hat to James Bach and Robert Sabourin)
• There is a difference between acting like a user and using software. (I tip my hat to James Whittaker).  The only way to truly measure software quality is to use the software, as a user.  How else can you hit just the right tests?
• Walking through the various test levels: Skilled testers act.  Team testers act (this is when non-testers on your product team help you test).  User acceptance testing is acting (they are still pretending to do some work).  Dogfooders…ah yes!  Dogfooders are finally not acting.  This is the first level to cross the boundary between pretending and acting.  Why not stop here?  It’s still weak in the infinite tests area.
• The next level is crowdsource testing.  Or is it crowdsourced testing?  I see both used frequently.  I can think of three ways to implement crowdsource testing:
• Use a crowdsource testing company.
• Do a limited beta release.
• Do a public beta release.
• Is crowdsource testing acting or using? If you outsource your crowdsource testing to a company (e.g., uTest, Mob4hire, TopCoder), now you’re hiring actors again.  However, if you do a limited or public beta release, you’re asking people to use your product.  See the difference?
• Beta…is a magic word.  It’s like stamping “draft” on the report you’re about to give your boss.  It also says, this product is alive!  New!  Exciting!  Maybe cutting edge.  It’s fixable still.  We won’t hang you out to dry.  It can only get better!
• Who should facilitate crowdsource testing?  The skilled in-house tester. Maybe crowdsource testing is just another tool in their toolbox.  If they spent this much time executing their own tests…
  
  
  
  …maybe instead, they could spend this much time (blue) executing their own tests, and this much time (yellow) collecting test information from their crowd…
  
  
  
• How do we get feedback from the crowd?  (from least cool to coolest) The crowd calls our tech support team, virtual feedback meetings, we go find the feedback on twitter etc., the crowd provides feedback via tightly integrated feedback options on the product’s UI, we programmatically collect it in a stealthy way.
• How do we enlist the crowd?  Via creative spin and marketing (e.g., follow Hipster’s example).
• Which products do we crowdsource test?  NOT the mission critical ones.  We crowdsource the trivial products like entertainment or social products.
• How far can we take this?  Far.  We can ask the crowd to build the product too.
• What can we do now?
• BA’s: determine beta candidate features.
• Testers: spend less time testing trivial features.  Outsource those to the crowd.
• Programmers: Decouple trivial features from critical features.
• UX Designers: Help users find beta features and provide feedback.
• Managers: Encourage more features to get released earlier.
• Executives: Hire a crowdsource test czar.

There it is in a nutshell.  Of course, the live presentation includes a bunch of fun extra stuff like the 1978 Alpo commercial that may have inspired the term “dogfooding”, as well as other theories and silliness.