If you find an escape (i.e., a bug for something marked “Done”), you may want to develop an automated check for it.  In a meeting today, there was a discussion about when the automated check needed to be developed?  Someone asked, “Should we put a task on the product backlog?”.  IMO:
The automated check should be developed when the bug fix is developed.  It should be part of the “Done” criteria for the bug.
Apply the above heuristically.  If your bug gets deferred to a future Sprint, deffer the automated check to that future Sprint.  If your bug gets fixed in the current Sprint, develop your automated check in the current Sprint.

My data warehouse team is adopting automated checking.  Along the way, we are discovering some doubters.  Doubters are a good problem.  They challenge us to make sure automation is appropriate.  In an upcoming meeting, we will try to answer the question in this blog post title.

My short answer:  Yes.

My long answer:  See below.

The following are data warehouse (or database) specific:

• More suited to machines – Machines are better than humans at examining lots of data quickly. 
• Not mentally stimulating for humans – (this is the other side of the above reason) Manual DB testers are hard to find.  Testers tend to like front-ends so they gravitate toward app dev teams.  DB testers need technical skills (e.g., DB dev skills).  People who have them prefer to do DB dev work.
• Straight forward repeatable automation patterns – For each new dimension table, we normally want the same types of automated checks.  This makes automated check design easier and faster to code.  The entire DW automation suite contains a smaller amount of design patterns than the average appliction.

The following are not limited to data warehouse (or database):

• Time to market – (Automated checks) help you go faster.  Randy Shoup says it well at 9:55 in this talk.  Writing quick and dirty software leads to technical debt which leads to no time to do it right (“technical debt viscious cycle”).  Writing automated checks as you write software  leads to a solid foundation which leads to confidence which leads to faster and better (“virtuous cycle of quality”)…Randy’s words.
• Regression checking - In general, machines are better than humans at indicating something changed. 
• Get the most from your human testing - Free the humans to focus on deep testing of new features, not shallow testing of old features.
• In case the business ever changes their mind - If you ever have to revist code to make changes or refactor, automated checks will help you do it quicker.  If you think the business will never change their mind, then maybe automation is not as important.
• Automated checks help document current functionality.
• Easier to fix problems - Automated checks triggered in a Continuous Integration find problems right after code is checked in.  These problems are usually easier to fix when fresh in a developer’s mind.

I read A Context-Driven Approach to Automation in Testing at the gym this morning.  I expected the authors to hate on automation but they didn’t.  Bravo.

They contrasted the (popular) perception that automation is cheap and easy because you don’t have to pay the computer, with the (not so popular) perception that automation requires a skilled human to design, code, maintain, and interpret the results of the automation.  That human also wants a paycheck.

I had a second scenario this week that gave me pause before resulting in the above practice.

ProductA is developed and maintained by ScrumTeamA, who writes automated checks for all User Stories and runs the checks in a CI.  ProductB is developed and maintained by ScrumTeamB.

ScrumTeamB developed UserStoryB, which required new code for both ProductA and ProductB.  ScrumTeamB merged the new product code into ProductA…but did NOT merge new test code to ProductA.  Now we have a problem.  Do you see it?

When ProductA deploys, how can we be sure the dependencies for UserStoyB are included?  All new product code for ProductA should probably be accompanied with new test code, regardless of the Scrum Team making the change.

The same practice might be suggested in environments without automation.  In other words, ScrumTeamB should probably give manual test scripts, lists, test fragments, or do knowledge transfer such that manual testers responsible for ProductA (i.e., ScrumTeamA) can perform the testing for UserStoryB prior to ProductA deployments.

…It seems obvious until you deal with integration tests and products with no automation.  I got tripped up by this example:

ProductA calls ProductB’s service, ServiceB.  Both products are owned by the same dev shop.  ServiceB keeps breaking in production, disrupting ProductA. ProductA has automated checks.  ProductB does NOT have automated checks.  Automated checks for ServiceB might help. Where would the automated checks for ServiceB live?

It’s tempting to say ProductA because ProductA has an automation framework with its automated checks running in a Continuous Integration on merge-to-dev.  It would be much quicker to add said automated checks to ProductA than ProductB.  However, said checks wouldn’t help b/c they would run in ProductA’s CI.  ProductB could still deploy to production with a broken ServiceB.

My lesson learned: Despite the ease of adding a check to ProductA’s CI, the check needs to be coupled with ProductB. 

In my case, until we invest in test automation for ProductB, said check(s) for ServiceB will be checks performed by humans.

While helping some testers, new to automation, I found myself in the unexpected position of trying to sell them on the idea that all test methods should be mutually exclusive.  Meaning, no automated check should depend on any other automated check…automated checks can run in any order…you can run them all, in any order, or you can run just one.
If I could take one test automation rule to my grave, this would be it.  I had forgotten that it was optional.
I know, I know, it’s seems so tempting to break this rule at first;  TestA puts the product-under-test in the perfect state for TestB.  Please don’t fall into this trap. 
Here are some reasons (I can think of) to keep your tests mutually exclusive:

• The Domino Effect – If TestB depends on TestA, and TestA fails, there is a good change TestB will fail, but not because the functionality TestB is checking fails.  And so on.
• Making a Check Mix – Once you have a good number of automated checks, you’ll want the freedom to break them into various suites.  You may want a smoke test suite, a regression test suite, a root check for a performance test, or other test missions that require only a handful of checks...dependencies will not allow this.
• Authoring – While coding an automated check (a new check or updating a check), you will want to execute that check over and over, without having to execute the whole suite.
• Easily Readable – When you review your automation coverage with your development team or stakeholders, you’ll want readable test methods.  That usually means each test method’s setup is clear.  Everything needed to understand that test method is contained within the scope of the test method.

Whilst searching for ways to measure the value of test automation, I read Doug Hoffman’s (I’m sure classic) Cost Benefits Analysis of Test Automation paper.

The first several pages were great.  He discussed intangibles that should be left out of an ROI calculation like an immediate reduction in perceived productivity of the test organization as the automation is first developed.  He went on to list the falsely expended benefits like the automation of existing manual tests.  Then he compared fixed automation costs like scripting tools to variable automation costs like test maintenance. 

Finally, Doug got to the formulas.  After careful analysis of some 30+ factors, one can start calculating automation ROI and efficiency benefits.  I rubbed my hands together and excitedly turned the page.  Then…I think I puked into my mouth a little as I saw the following:

In the end, I latched onto one powerful statement Doug made, almost in passing, he said:

If the benefits of automation are required, then the ROI computation is unnecessary, the investment is required…it’s an expense.

If the benefits include reducing risk by performing automated checking that would not be possible by humans (e.g., complex math, millions of comparisons, diffs, load, performance, precision), then say no more…

I don’t want to suffer through the computations.

Now that my engineering team is automating beyond the unit test level, this question comes up daily.  I wish there were an easy answer.

If we make a distinction between checking and testing, no “tests” should be automated.  The question instead becomes, which “checks” should be automated?  Let’s go with that.

I’ll tell you what I think below, ranking the more important at the top:

Consider automating checks when they…

1. can only feasibly be checked by a machine (e.g., complex math, millions of comparisons, diffs, load, performance, precision).  These are checks machines do better than humans.
2. are important.  Do we need to execute them prior to each build, deployment, or release?  This list of checks will grow over time.  The cost of not automating is less time for the “testing” that helps us learn new information.
3. can be automated below the presentation layer.  Automating checks at the API layer is considerably less expensive than at the UI layer.  The automated checks will provide faster feedback and be less brittle.
4. will be repeated frequently.  A simplified decision: Is the time it takes a human to program, maintain, execute, and interpret the automated check’s results over time (e.g., 2 years), less than the time it takes a human to perform said check over the same time span (e.g., 2 years).  This overlaps with #2.
5. check something that is at risk of failing.  Do we frequently break things when we change this module?
6. are feasible to automate.  We can’t sufficiently automate tests for things like usability and charisma.
7. are requested by my project team.  Have a discussion with your project team about which checks to automate.  Weigh your targets with what your team thinks should be automated.
8. can be automated using existing frameworks and patterns.  It is much cheaper to automate the types of checks you’ve already successfully automated.
9. are checks.  “Checks” are mundane for humans to perform.  “Tests” are not because they are different each time. 

What am I missing?

Most of the testers at my new company do not have programming skills (or at least are not putting them to use).  This is not necessarily a bad thing.  But in our case, many of the products-under-test are perfect candidates for automation (e.g., they are API rich).

We are going through an Agile transformation.  Discussions about tying programmatic checks to “Done” criteria are occurring and most testers are now interested in getting involved with automation.  But how?

I think this is a common challenge.

Here are some ways I have had success getting manual testers involved in automation.  I’ll start with the easiest and work my way down to those requiring more ambition.  A tester wanting to get involved in automation can:

1. Do unit test reviews with their programmers.  Ask the programmers to walk you through the unit tests.  If you get lost ask questions like, “what would cause this unit test to fail?” or “can you explain the purpose of this test at a domain level?”.
2. Work with automators to inform the checks they automate.  If you have people focused on writing automated checks, help them determine what automation might help you.  Which checks do you often repeat?  Which are boring?
3. Design/request a test utility that mocks some crucial interface or makes the invisible visible.  Bounce ideas off your programmers and see if you can design test tools to speed things up.  This is not traditional automation.  But it is automation by some definitions.
4. Use data-driven automation to author/maintain important checks via a spreadsheet.  This is a brilliant approach because it lets the test automater focus on what they love, designing clever automation.  It lets the tester focus on what they love, designing clever inputs.  Show the tester where the spreadsheet is and how to kick off the automation.
5. Copy and paste an automated check pattern from an IDE, rename the check and change the inputs and expected results to create new checks.  This takes 0-to-little coding skills.  This is a potential end goal.  If a manual tester gets to this point, buy them a beer and don’t push them further.  This leads to a great deal of value, and going further can get awkward.
6. Follow an automated check pattern but extend the framework.  Spend some time outside of work learning to code. 
7. Stand up an automation framework, design automated checks.  Support an Agile team by programming all necessary automated checks.  Spend extensive personal time learning to code.  Read books, write personal programs, take online courses, find a mentor. 

If you read part 1, you may be wondering how my automated check performed…

The programmer deployed the seeded bug and I’m happy to report, my automated check found it in 28 seconds! 

Afterwards, he seeded two additional bugs.  The automated check found those as well.  I had to temporarily modify the automated check code to ignore the first bug in order to find the second.  This is because the check stops checking as soon as it finds one problem.  I could tweak the code to collect problems and keep checking but I prefer the current design.

Here is the high level generic design of said check:

Build the golden masters:

• Make scalable checks - Before test execution, build multiple golden masters per coverage ambition.  This is a one-time-only task (until the golden masters need to be updated per expected changes).
• Bypass GUI when possible – Each of my golden masters consist of the response XML from a web service call, saved to a file.  Each XML response has over a half a million nodes, which are mapped to a complex GUI.  In my case, my automated check will bypass the GUI.  GUI automation could never have found the above seeded bug in 28 seconds.  My product-under-test takes about 1.5 minutes just to log in and navigate to the module being tested. Waiting for the GUI to refresh after the countless service calls being made in the automated check would have taken hours.
• Golden masters must be golden! Use a known good source for the service call.  I used Production because my downstream environments are populated with data restored from production.  You could use a test environment as long as it was in a known good state.
• Use static data - Build the golden masters using service request parameters that return a static response.  In other words, when I call said service in the future, I want the same data returned.  I used service request parameters to pull historical data because I expect it to be the same data next week, month, year, etc.
• Automate golden master building - I wrote a utility method to build my golden masters.  This is basically re-used code from the test method, which builds the new objects to compare to the golden masters.

Do some testing:

• Compare - This is the test method.  It calls the code-under-test using the same service request parameters used to build the golden masters.  The XML service response from the code-under-test is then compared to that of the archived golden masters, line-by-line.
• Ignore expected changes - In my case there are some XML nodes the check ignores.  These are nodes with values I expect to differ.  For example, the CreatedDate node of the service response object will always be different from that of the golden master.
• Report - If any non-ignored XML line is different, it’s probably a bug, fail the automated check, report the differences with line number and file (see below) references and investigate.
• Write Files - For my goals, I have 11 different golden masters (to compare with 11 distinct service response objects).  The automated check loops through all 11 golden master scenarios, writing each service response XML to a file.  The automated check doesn’t use the files, they are there for me.  This gives me the option to manually compare suspect new files to golden masters with a diff tool, an effective way of investigating bugs and determining patterns.

I’m feeling quite cocky at the moment.  So cocky that I just asked my lead programmers to secretly insert a bug into the most complex area of the system under test.

Having just finished another epic automated check based on the Golden Master approach I discussed earlier, and seeing that most of the team is on pre-Thanksgiving vacation, this is the perfect time to “seed a bug”.  Theoretically, this new automated check should help put our largest source of regression bugs to rest and I am going to test it.

The programmer says he will hide the needle in the haystack by tomorrow.

I’m waiting…

You’ve got this new thing to test. 

You just read about a tester who used Selenium and he looked pretty cool in his bio picture.  Come on, you could do that.  You could write an automated check for this.  As you start coding, you realize your initial vision was too ambitious so you revise it.  Even with the revised design you’re running into problems with the test stack.  You may not be able to automate the initial checks you wanted, but you can automate this other thing.  That’s something.  Besides, this is fun.  The end is in sight.  It will be so satisfying to solve this.  You need some tasks with closure in your job, right?  This automated check has a clear output.  You’ve almost cracked this thing…cut another corner and it just might work.  Success!  The test passes!  You see green!  You rule!  You’re the Henry Ford of testing!  You should wear a cape to work!

Now that your automated thingamajig is working and bug free, you can finally get back to what you were going to test.  Now what was it?

I’m not hating on test automation.  I’m just reminding myself of its intoxicating trap.  Keep your eyes open.

“Golden Master”, it sounds like the bad guy in a James Bond movie.  I first heard the term used by Doug Hoffman at STPCon Spring 2012 during his Exploratory Test Automation workshop.  Lately, I’ve been writing automated golden master tests that check hundreds of things with very little test code.

I think Golden-Master-Based testing is super powerful, especially when paired with automation.

A golden master is simply a known good version of something from your product-under-test.  It might be a:

• web page
• reference table
• grid populated with values
• report
• or some other file output by your product

Production is an excellent place to find golden masters because if users are using it, it’s probably correct.  But golden masters can also be fabricated by a tester.

Let’s say your product outputs an invoice file.  Here’s a powerful regression test in three steps:

1. Capture a known good invoice file from production (or a QA environment).  This file is your golden master. 
2. Using the same parameters that were used to create the golden master, re-create the invoice file on the new code under test. 
3. Programmatically compare the new invoice to the golden master using your favorite diff tool or code.

Tips and Ideas:

• Make sure the risky business logic code you want to test is being exercised.
• If you expand on this test, and fully automate it, account for differences you don’t care about (e.g., the invoice generated date in the footer, new features you are expecting to not yet be in production). 
• Make it a data-driven test. Pass in a list of orders and customers, retrieve production golden masters and compare them to dynamically generated versions based on the new code.
• Use interesting dates and customers.  Iterate through thousands of scenarios using that same automation code.
• Use examples from the past that may not be subject to changes after capturing the golden master.
• Structure your tests assertions to help interpret failures.  The first assertion on the invoice file might be, does the item line count match?  The second might be, do each line’s values match?
• Get creative.  Golden masters can be nearly anything.

Who else uses this approach?  I would love to hear your examples.

I often hear people describe their automated test approach by naming the tool, framework, harness, technology, test runner, or structure/format.  I’ve described mine the same way.  It’s safe.  It’s simple.  It’s established.  “We use Cucumber”.

Lately, I’ve seen things differently.

Instead of trying to pigeon hole each automated check into a tightly controlled format for an entire project, why not design automated checks for each Story, based on their best fit for that story?

I think this notion comes from my context-driven test schooling.  Here’s an example:

On my current project, we said “let’s write BDD-style automated checks”.  We found it awkward to pigeon-hole many of our checks into Given, When, Then.  After eventually dropping the mandate for BDD-style, I discovered the not-as-natural-language style to be easier to read, more flexible, and quicker to author…for some Stories.  Some Stories are good candidates for data-driven checks authored via Excel.  Some might require manual testing with a mocked product...computer-assisted-exploratory-testing…another use of automation.  Other Stories might test better using non-deterministic automated diffs.

Sandboxing all your automated checks into FitNesse might make test execution easier.  But it might stifle test innovation.

Dear Test Automators,

The next time you discuss automation results, please consider qualifying the context of the word “bug”.

If automation fails, it means one of two things:

1. There is a bug in the product-under-test.
2. There is a bug in the automation.

The former is waaaaaay more important than the latter.  Maybe not to you, but certainly for your audience.

Instead of saying,

“This automated check failed”,

consider saying,

“This automated check failed because of a bug in the product-under-test”.

Instead of saying,

“I’m working on a bug”,

consider saying,

“I’m working on a bug in the automation”.

Your world is arguably more complex than that of testers who don’t use automation.  You must test twice as many programs (the automation and the product-under-test).  Please consider being precise when you communicate.

I heard a great interview with performance tester, Scott Barber.  Two things Scott said stayed with me.  Here is the first.

Automated checks that record a time span (e.g., existing automated checks hijacked to become performance tests) may not need to result in Pass/Fail, as respect to performance.  Instead, they could just collect their time span result as data points.  These data points can help identify patterns:

• Maybe the time span increases by 2 seconds after each new build.
• Maybe the time span increases by 2 seconds after each test run on the same build.
• Maybe the time span unexpectedly decreases after a build.
• etc.

My System 1 thinking tells me to add a performance threshold that resolves automated checks to a mere Pass/Fail.  Had I done that, I would have missed the full story, as Facebook did. 

Rumor has it, Facebook had a significant production performance bug that resulted from reliance on a performance test that didn’t report performance increases.  It was supposed to Fail if the performance dropped.

At any rate, I can certainly see the advantage of dropping Pass/Fail in some cases and forcing yourself to analyze collected data points instead.

I often hear skeptics question the value of test automation.  Their questioning is healthy for the test industry and it might flush out bad test automation.  I hope it continues.

But shouldn’t these same questions be raised about human testing (AKA Manual testing)?  If these same skeptics judged human testing with the same level of scrutiny, might it improve human testing? 

First, the common criticisms of test automation:

• Sure, you have a lot of automated checks in your automated regression check suite, but how many actually find bugs?
• It would take hours to write an automated check for that.  A human could test it in a few seconds.
• Automated checks can’t adapt to minor changes in the system under test.  Therefore, the automated checks break all the time.
• We never get the ROI we expect with test automation.  Plus, it’s difficult to measure ROI for test automation.
• We don’t need test automation.  Our manual testers appear to be doing just fine.

Now let’s turn them around to question manual testing:

• Sure, you have a lot of manual tests in your manual regression test suite, but how many actually find bugs?
• It would take hours for a human to test that.  A machine could test it in a few seconds.
• Manual testers are good at adapting to minor changes in the system under test.  Sometimes, they aren’t even aware of their adaptions.  Therefore, manual testers often miss important problems.
• We never get the ROI we expected with manual testing.  Plus, it’s difficult to measure ROI for manual testing.
• We don’t need manual testers.  Our programmers appear to be doing just fine with testing.

During a recent exchange about the value of automated checks, someone rhetorically asked:

“Is automation about finding lots of bugs or triggering investigation?”

Well…the later, right?

• When an automated check passes consistently for months then suddenly fails, it’s an indication the system-under-test (SUT) probably unexpectedly changed.  Investigate!  The SUT change may not be directly related to the check but who cares, you can still pat the check on the back and say, “thank you automated check, for warning me about the SUT change”.
• When you design/code an automated check, you are learning how to interact with your SUT and investigating it.  If there are bugs uncovered during the automated check design/coding, you report them now and assume the automated checks should happily PASS for the rest of their existence.
• If someone is organized enough to tell you the SUT is about to change, you should test the change and assess the impact on your automated checks and make necessary updates.  Doing so requires investigating said SUT changes.

In conclusion, one can argue, even the lamest of automated checks can still provide value.  Then again, one can argue most anything.

Here’s another failure story, per the post where I complained about people not telling enough test failure stories.

Years ago, after learning about Keyword-Driven Automation, I wrote an automation framework called OKRA (Object Keyword-Driven Repository for Automation).  @Wiggly came up with the name.  Each automated check was written as a separate Excel worksheet, using dynamic dropdowns to select from available Action and Object keywords in Excel.  The driver was written in VBScript via QTP.  It worked, for a little while, however:

• One Automator (me) could not keep up with 16 programmers.  The checks quickly became too old to matter.  FAIL!
• An Automator with little formal programming training, writing half-ass code with VBScript, could not get help from a team of C# focused programmers.  FAIL!
• The product under test was a .Net Winforms app full of important drag-n-drop functionality, sitting on top of constantly changing, time-sensitive, data.  Testability was never considered.  FAIL!
• OKRA was completely UI-based automation.  FAIL!

Later, a product programmer took an interest in developing his own automation framework. It would allow manual testers to write automated checks by building visual workflows.  This was a Microsoft technology called MS Workflow or something like that.  The programmer worked in his spare time over the course of about a year.  It eventually faded into oblivion and was never introduced to testers.  FAIL!

Finally, I hired a real automator, with solid programming skills, and attempted to give it another try.  This time we picked Microsoft’s recently launched CodedUI framework and wrote the tests in C# so the product programmers could collaborate.  I stood in front of my SVP and project team and declared,

“This automation will shave 2 days off our regression test effort each iteration!”

However:

• The automator was often responsible for writing automated checks for a product they barely understood.  FAIL!
• Despite the fact that CodedUI was marketed by Microsoft as being the best automation framework for .Net Winform apps, it failed to quickly identify most UI objects, especially for 3rd party controls.
• Although, at first, I pushed for significant amounts of automation below the presentation layer, the automator focused more energy on UI automation.  I eventually gave in too.  The tests were slow at best and human testers could not afford to wait.  FAIL!  Note: this was not the automators failure, it was my poor direction.

At this point, I’ve given up all efforts to automate this beast of an application.

Can you relate?

Have you ever been to a restaurant with a kitchen window?  Well, sometimes it may be best not to show the customers what the chicken looks like until it is served.

A tester on my team has something similar to a kitchen window for his automated checks; the results are available to the project team.

Here’s the rub: 

His new automated check scenario batches are likely to result in…say, a 10% failure rate (e.g., 17 failed checks).  These failures are typically bugs in the automated checks, not the product under test.  Note: this project only has one environment at this point.

When a good curious product owner looks through the kitchen window and sees 17 Failures, it can be scary!  Are these product bugs?  Are these temporary failures? 

Here’s how we solved this little problem:

• Most of the time, we close the curtains.  The tester writes new automated checks in a sandbox, debugs them, then merges them to a public list.
• When the curtains are open, we are careful to explain, “this chicken is not yet ready to eat”.  We added an “Ignore” attribute to the checks so they can be filtered from sight.