I had the pleasure of eating lunch with Dorothy Graham at Stareast. Dorothy is the coauthor of “Software Test Automation”, which has been a well respected book on the subject for the last 10 years. A colleague recently referred me to a great article, “That’s No Reason to Automate!”, coauthored by Dorothy in the current issue of Better Software.

In the article, Dorothy debunks many popular objectives used for test automation and suggests more reasonable versions of each. This article is helping me wrap my brain around my own test automation objectives (I just hired a test automator) but it was also just great to hear a recognized test automation expert empower manual testers so much.

I'll paraphrase/quote some sentences that caught my attention and some of the (needs improvement) test automation objectives they contradict.

Objective: Automation should find more bugs.

• “Good testing is not found in the number of tests run, but in the value of the tests that are run.”
• The factor that determines if more bugs will be found is the quality of the tests, not the quantity. Per Dorothy, “It is the testing that finds bugs – not the automation”. The trick is to free up the tester’s time so they can find more bugs. This may be achieved by using automation to execute the mundane tests (that probably won’t find more bugs).
  

Objective: Automation should reduce testing staff.

• More staff are typically need to incorporate test automation. People with test script development skills will need to be added, in addition to people with testing skills.
• Automation supports testing activities but does not replace them. Test tools cannot make intelligent decisions about which tests to run and when, nor can they analyze results and investigate problems.
  

Objective: Automation should reduce testing time.

• “The main thing that causes increased testing time is the quality of the software – the number of bugs that are already there…the quality of the software is the responsibility of the developers, not the testers or the test automators”

Objective: Automation should allow us to run more tests to get more coverage.

• A count of the number of automated tests is a useless way of gauging the contribution of automation to testing. If the test team ends up with a set of tests that are hardly ever run by the testers, that is not the fault of the test automators. That is the fault of the testers for choosing the wrong tests to automate.

Objective: We should automate X% of our tests.

• Automating 2% of your most important tests could be better than automating 50% of your tests that don’t provide value.

Note: The article and book were co-authored by both Dorothy Graham and Mark Fewster. Although I did not have lunch with Mark, I'm sure he is a great guy too!