During a recent exchange about the value of automated checks, someone rhetorically asked:
“Is automation about finding lots of bugs or triggering investigation?”
Well…the later, right?
- When an automated check passes consistently for months then suddenly fails, it’s an indication the system-under-test (SUT) probably unexpectedly changed. Investigate! The SUT change may not be directly related to the check but who cares, you can still pat the check on the back and say, “thank you automated check, for warning me about the SUT change”.
- When you design/code an automated check, you are learning how to interact with your SUT and investigating it. If there are bugs uncovered during the automated check design/coding, you report them now and assume the automated checks should happily PASS for the rest of their existence.
- If someone is organized enough to tell you the SUT is about to change, you should test the change and assess the impact on your automated checks and make necessary updates. Doing so requires investigating said SUT changes.
In conclusion, one can argue, even the lamest of automated checks can still provide value. Then again, one can argue most anything.