I often hear skeptics question the value of test automation.  Their questioning is healthy for the test industry and it might flush out bad test automation.  I hope it continues.

But shouldn’t these same questions be raised about human testing (AKA Manual testing)?  If these same skeptics judged human testing with the same level of scrutiny, might it improve human testing? 

First, the common criticisms of test automation:

• Sure, you have a lot of automated checks in your automated regression check suite, but how many actually find bugs?
• It would take hours to write an automated check for that.  A human could test it in a few seconds.
• Automated checks can’t adapt to minor changes in the system under test.  Therefore, the automated checks break all the time.
• We never get the ROI we expect with test automation.  Plus, it’s difficult to measure ROI for test automation.
• We don’t need test automation.  Our manual testers appear to be doing just fine.

Now let’s turn them around to question manual testing:

• Sure, you have a lot of manual tests in your manual regression test suite, but how many actually find bugs?
• It would take hours for a human to test that.  A machine could test it in a few seconds.
• Manual testers are good at adapting to minor changes in the system under test.  Sometimes, they aren’t even aware of their adaptions.  Therefore, manual testers often miss important problems.
• We never get the ROI we expected with manual testing.  Plus, it’s difficult to measure ROI for manual testing.
• We don’t need manual testers.  Our programmers appear to be doing just fine with testing.

It’s true.  Our job rocks.  Huff Post called it the 2nd happiest job in America this year.  Second only to being a DBA…yawn.  Two years ago, Forbes said testing was #1. 

But why?  Neither article goes in depth.  Maybe it’s because all news is good news, for a tester:

• The System Under Test (SUT) is crashing in QA, it doesn’t work, it’s a steaming pile of…YES!  My testing was valuable!  My life has meaning!  My testing just saved users from this nightmare!
• The SUT is finally working.  Awesome!  It’s so nice being part of a development team that can deliver quality software.  I can finally stop testing it and move on.  Our users are going to love this.

See?  Either way it’s good news.

Or maybe I just spin it that way to love my job more.  So be it.  If you think your testing job is stressful, you may want to make a few adjustments in how you work.  Read my You’re a Tester, Relax post.

During a recent exchange about the value of automated checks, someone rhetorically asked:

“Is automation about finding lots of bugs or triggering investigation?”

Well…the later, right?

• When an automated check passes consistently for months then suddenly fails, it’s an indication the system-under-test (SUT) probably unexpectedly changed.  Investigate!  The SUT change may not be directly related to the check but who cares, you can still pat the check on the back and say, “thank you automated check, for warning me about the SUT change”.
• When you design/code an automated check, you are learning how to interact with your SUT and investigating it.  If there are bugs uncovered during the automated check design/coding, you report them now and assume the automated checks should happily PASS for the rest of their existence.
• If someone is organized enough to tell you the SUT is about to change, you should test the change and assess the impact on your automated checks and make necessary updates.  Doing so requires investigating said SUT changes.

In conclusion, one can argue, even the lamest of automated checks can still provide value.  Then again, one can argue most anything.

Perhaps Jimmy John’s should have hired some software testers before slapping their “Order Online” logo all over the place.

Yesterday, while entering a group order online, I had a little trouble typing my Delivery Instructions in the “memo-size” text box.

The only way to add a drink to your sandwich order was to add a second sandwich.  Um, I only want one sandwich.

I selected the earliest delivery time available:

However, after painstakingly collecting the orders of about 17 people, when I submitted my group order, Jimmy John’s showed me this user validation message:

And prior to me clicking the OK button, Jimmy John’s cancelled my entire order, including sending this email to all 17 people :

There is some additional context here that is too complex for this post.  Suffice it to say, I was irritated that my order was cancelled with no warning.

I called the Jimmy John’s phone number provided on the top of my screen and the dude said, “we have no idea how to retrieve your order, we just get a printout when it’s submitted”. 

In the end, the good people at my Jimmy John’s franchise accepted a fax with screen captures of my original group order (we tried email, but they couldn’t retrieve it) and they delivered the order flawlessly.

Is there a name for this?  If not, I’m going to call it a “fire drill test”.

• A fire drill test would typically not be automated because it will probably only be used once.
• A fire drill test informs product design so it may be worth executing early.
• A fire drill test might be a good test candidate to delegated to a project team programmer.

Fire drill test examples:

• Our product ingests files from an ftp site daily.  What if the files are not available for three days?  Can our product catch up gracefully?
• Our product outputs a file to a shared directory.  What if someone removes write permission to the shared directory for our product?
• Our product uses a nightly job to process data?  If the nightly job fails due to off-hour server maintenance, how will we know?  How will we recover?
• Our product displays data from an external web service.   What happens if the web service is down?

Too often, us testers have so much functional testing to do, we overlook the non-functional testing or save it for the end.  If we give these non-functional tests a catchy name like “Fire Drill Test”, maybe it will help us remember them during test brainstorming.

I Attended John Stevenson’s great talk and workshop at Monday night’s Software Testing Club Atlanta.  I’m happy to report the meeting had about 15 in-person attendees and zero virtual attendees.  Maybe someone read my post.

John is a thoughtful and passionate tester.  He managed to hold our attention for 3 hours!  Here are the highlights from my notes:

• The human brain can store 3TBs of information; This is only 1 millionth of the new information released on the internet every day.
• Over stimulation leads to mental illness.
• John showed us a picture and asked what we saw.  We saw a tree, flowers, the sun, etc.  Then John told us the picture was randomly generated. The point?  People see patterns even when they don’t exist.  Presumably to make sense out of information overload.
• Don’t tell your testing stories with numbers.  “A statistician drowned while crossing a river with an average depth of 3 feet”; Isn’t that like, “99 percent of my tests passed”?
• Don’t be a tester that waits until testing “is done” to communicate the results.  Communicate the test results you collected today?  I love this and plan to blog about it.
• Testers, stop following the same routines.  Try doing something different.  You might end up discovering new information.
• Testers, stop hiding what you do.  Get better at transparency and explaining your testing.  Put your tests on a public wiki.
• Critical thinking takes practice.  It is a skill.
• “The Pause”. Huh?  Really?  So?  Great critical thinking model explained in brief here.
• A model for skepticism.  FiLCHeRS.
• If you challenge someone’s view, be aware of respecting it.
• Ways to deal with information overload:
• Slow down.
• Don’t over commit.
• Don’t fear mistakes.  But do learn from them.  This is how children learn.  Play.
• (Testing specific)  Make your testing commitments short so you can throw them away without losing much.  Don’t write some elaborate test that takes a week to write because it just might turn out to be the wrong test.
• You spend a 3rd of your life at work.  Figure out how to enjoy work.
• John led us through a series of group activities including the following:
• Playing Disruptus to practice creative thinking.  (i.e., playing Scamper.)
• Playing Story War to practice bug advocacy.
• Determining if the 5 test phases (Documentation, Planning, Execution, Analysis, Reporting) each use Creative Thinking or Critical thinking.
• Books John referenced that I would like to read:
• The Signal and the Noise – Nate Silver
• Thinking Fast and Slow – Daniel Kahneman
• You are Not So Smart – David McRaney

At this week’s metric themed Atlanta Scrum User’s Group meetup, I asked the audience if they knew of any metrics (that could not be gamed) that could trigger rewards for development teams.  The reaction was as if I had just praised Planned Parenthood at a Pro-life rally…everyone talking over each other to convince me I was wrong to even ask.

The facilitator later rewarded me with a door prize for the most controversial question.  What?

Maybe my development team and I are on a different planet than the Agile-istas I encountered last night.  Because we are currently doing what I proposed, and it doesn’t appear to be causing any harm.

Currently, if 135 story points are delivered in the prior month AND no showstopper production bugs were discovered, everyone on the team gets a free half-day-off to use as they see fit.  We’ve achieved it twice in the past year.  The most enthusiastic part of each retrospective is to observe the prior months metrics and determine if we reached our “stretch goal”.  It’s…fun.  Let me repeat that.  It’s actually fun to reward yourself for extraordinary work.

Last night’s question was part of a quest I’ve been on to find a better reward trigger.  Throughput and Quality is what we were aiming for.  And I think we’ve gotten close.  I would like to find a better metric than Velocity, however, because story point estimation is fuzzy.  If I could easily measure “customer delight”, I would.

At the meeting, I learned about the Class of Service metric.  And I’m mulling over the idea of suggesting a “Dev Forward” % stretch goal for a given time period.

But what is this nerve I keep touching about rewards for good work?

On weekends, when I perform an extraordinary task around the house like getting up on the roof to repair a leak, fixing an electrical issue, constructing built-in furniture to solve a space problem, finishing a particularly large batch of “Thank You” cards, or whatever…I like to reward myself with a beer, buying a new power tool, relaxing in front of the TV, taking a long hot shower, etc.

Rewards rock.  What’s wrong with treating ourselves at work too?