“Golden Master”, it sounds like the bad guy in a James Bond movie. I first heard the term used by Doug Hoffman at STPCon Spring 2012 during his Exploratory Test Automation workshop. Lately, I’ve been writing automated golden master tests that check hundreds of things with very little test code.
I think Golden-Master-Based testing is super powerful, especially when paired with automation.
A golden master is simply a known good version of something from your product-under-test. It might be a:
- web page
- reference table
- grid populated with values
- or some other file output by your product
Production is an excellent place to find golden masters because if users are using it, it’s probably correct. But golden masters can also be fabricated by a tester.
Let’s say your product outputs an invoice file. Here’s a powerful regression test in three steps:
- Capture a known good invoice file from production (or a QA environment). This file is your golden master.
- Using the same parameters that were used to create the golden master, re-create the invoice file on the new code under test.
- Programmatically compare the new invoice to the golden master using your favorite diff tool or code.
Tips and Ideas:
- Make sure the risky business logic code you want to test is being exercised.
- If you expand on this test, and fully automate it, account for differences you don’t care about (e.g., the invoice generated date in the footer, new features you are expecting to not yet be in production).
- Make it a data-driven test. Pass in a list of orders and customers, retrieve production golden masters and compare them to dynamically generated versions based on the new code.
- Use interesting dates and customers. Iterate through thousands of scenarios using that same automation code.
- Use examples from the past that may not be subject to changes after capturing the golden master.
- Structure your tests assertions to help interpret failures. The first assertion on the invoice file might be, does the item line count match? The second might be, do each line’s values match?
- Get creative. Golden masters can be nearly anything.
Who else uses this approach? I would love to hear your examples.