I’m not asking if they *can* run unattended.  I’m asking if they do run unattended…consistently…without ever failing to start, hanging, or requiring any human intervention whatsoever…EVER.
Automators, be careful.  If you tell too many stories about unattended check-suite runs, the non-automators just might start believing you.  And guess what will happen if they start running your checks?  You know that sound when Pac-Man dies, that’s what they’ll think of your automated checks.
I remember hearing a QA Director attempt to encourage “test automation” by telling fantastical stories of his tester past:
“We used to kick off our automated tests at 2PM and then go home for the day.  The next day, we would just look at the execution results and be done.”
Years later, I’ve learned to be cynical about said stories.  In fact, I have yet to see an automated test suite (including my own) that consistently runs without ever requiring the slightest intervention from humans, who unknowingly may:

  • Prep the test environment “just right” before clicking “Run”.
  • Restart the suite when it hangs and hope the anomaly goes away.
  • Re-run the failed checks because they normally pass on the next attempt.
  • Realize the suite works better when kicked off in smaller chunks.
  • Recognize that sweet spot, between server maintenance windows, where the checks have a history of happily running without hardware interruptions.
IMO, it’s not a problem if the automator has to periodically do one or more of the above.  It’s only a problem if we, as automators, spread untruths about the real effort behind our automated checks.

2 comments:

  1. hot roller press said...
    This comment has been removed by a blog administrator.
  2. Cheezy said...

    Eric,

    Interesting post but it doesn't match my overall experience. The problems you describe are typical for teams just getting started with test automation due to problems with their test suites (e.g. they do not have test data management built into their tests) but I rarely find these items with teams that have been doing it for a while. Perhaps we can compare notes at StarCanada.

    -Cheezy



Copyright 2006| Blogger Templates by GeckoandFly modified and converted to Blogger Beta by Blogcrowds.
No part of the content or the blog may be reproduced without prior written permission.