Now that my engineering team is automating beyond the unit test level, this question comes up daily.  I wish there were an easy answer.

If we make a distinction between checking and testing, no “tests” should be automated.  The question instead becomes, which “checks” should be automated?  Let’s go with that.

I’ll tell you what I think below, ranking the more important at the top:

Consider automating checks when they…

  1. can only feasibly be checked by a machine (e.g., complex math, millions of comparisons, diffs, load, performance, precision).  These are checks machines do better than humans.
  2. are important.  Do we need to execute them prior to each build, deployment, or release?  This list of checks will grow over time.  The cost of not automating is less time for the “testing” that helps us learn new information.
  3. can be automated below the presentation layer.  Automating checks at the API layer is considerably less expensive than at the UI layer.  The automated checks will provide faster feedback and be less brittle.
  4. will be repeated frequently.  A simplified decision: Is the time it takes a human to program, maintain, execute, and interpret the automated check’s results over time (e.g., 2 years), less than the time it takes a human to perform said check over the same time span (e.g., 2 years).  This overlaps with #2.
  5. check something that is at risk of failing.  Do we frequently break things when we change this module?
  6. are feasible to automate.  We can’t sufficiently automate tests for things like usability and charisma.
  7. are requested by my project team.  Have a discussion with your project team about which checks to automate.  Weigh your targets with what your team thinks should be automated.
  8. can be automated using existing frameworks and patterns.  It is much cheaper to automate the types of checks you’ve already successfully automated.
  9. are checks.  “Checks” are mundane for humans to perform.  “Tests” are not because they are different each time. 

 

What am I missing?

6 comments:

  1. Susovan said...

    Thanks Eric for this article. I have 1 confusion. How we can identify Regression test cases from Functional test cases in Agile project with releases after every 2 months.

  2. Eric Jacobson said...

    Here is how I currently see things, Susovan:

    A subset of your Functional tests will probably become regression checks. Use the suggestions in my post to determine which to automate and which to repeat as manual tests. Repeat the automated regression checks as often as possible (e.g., at each product code check-in). Don't wait until right before the Release to regression test, even for your manual tests. Try to do it in parallel with new Functional testing.

    Always run all your automated checks for regression testing. As your manual test/check list increases, it's up to the testers to decide which to spend their precious time on. And that is a different discussion!

  3. Mona said...

    Hi Eric,

    Thanks for the great blogs. I did read your blog about JART. Would you mind sending it to me please? I promise to tell everybody how good coder you are.

    Thanks,
    Mona

  4. Eric Jacobson said...

    Mona, use the email link on the right of my blog to send me your contact info and we can talk.

  5. Unknown said...

    Its very interesting to know the exact features which are to be considered while automating test cases. Nice article.

  6. Anonymous said...

    Hi Eric,
    On my opinion, automated should be when test has a larger dimension. For instance - if during the test we need monitoring states of few dozens or hundreds parameters.
    Best regards, Alex.



Copyright 2006| Blogger Templates by GeckoandFly modified and converted to Blogger Beta by Blogcrowds.
No part of the content or the blog may be reproduced without prior written permission.