When bugs escape to production, does your team adjust?
We started using the following model on one of my projects. It appears to work fairly well. Every 60 days we meet and review the list of “escapes” (i.e., bugs found in production). For each escape, we ask the following questions:
- Could we do something to catch bugs of this nature?
- Is it worth the extra effort?
- If so, who will be responsible for said effort?
The answer to #1 is typically “yes”. Creative people are good at imagining ultimate testing. It’s especially easy when you already know the bug. There are some exceptions though. Some escapes can only be caught in production (e.g., a portion of our project is developed in production and has no test environment).
The answer to #2 is split between “yes” and “no”. We may say “yes” if the bug has escaped more than once, significantly impacts users, or when the extra effort is manageable. We may say “no” when a mechanism is in place to alert our team of the prod error; we can patch some of these escapes before they affect users, with less effort than required to catch them in non-prod environments.
The answer to #3 falls to Testers, Programmers, BAs, and sometimes both or all.
So…when bugs escape to production, does my team adjust? Sometimes.