On a production support kanban development team, a process dilema came up. In the case where something needs to be tested by a tester:
- Should the tester perform the testing first in a development environment, then in a production-like environment after the thing-under-test has been packaged and deployed? Note: in this case, the package/deploy process is handled semi-manually by two separate teams, so there is a delay.
- Or, should the tester perform all the testing in a production-like environment after the thing-under-test has been packaged and deployed?
Advantage of scenario 1 above:
- Dev environment testing shortens the feedback loop. This would be deep testing. If problems surface they would be quicker and less risky to fix. The post-package testing would be shallow testing, answering questions like: did the stuff I deep tested get deployed properly?
Advantage of scenario 2 above:
- Knock out the testing in one environment. The deep testing will indirectly cover the package/deployment testing.
From the surface, scenario 2 looks better because it only requires one testing chunk, NOT two chunks separated by a lengthy gap. But what happens if a problem surfaces in scenario 2? Now we must go through two lengthy gaps. How about a third problem? Three gaps. And so on.
My conclusion: Scenario 1 is better unless this type of thing-under-test is easy and has a history of zero problems.