“I’m just the tester, if it doesn’t run it’s not my problem, it’s the deployment team’s problem. I can tell you how well it will work, but first you’ve got to deploy it properly.”

One of the most difficult problems to prevent is a configuration problem; a setting that is specific to production.  You can attempt perfect testing in a non-production environment, but as soon as your Config Management guys roll it out to prod with the prod config settings, the best you can do is cross your fingers (unless you’re able to test in prod).

After a recent prod server migration, my config management guys got stuck scrambling around trying to fix various prod config problems.  We had all tested the deployment scripts in multiple non-prod environments.  But it still didn’t prepare us for the real thing.

It’s too late for testers to help now.

I’ve been asking myself what I could have done differently.  The answer seems to be, asking/executing more hypothetical questions/tests, like:

  • If this scheduled nightly task fails to execute, how will we know?
  • If this scheduled nightly task fails to execute, how will we recover?

But often I skip the above because I’m so focused on:

  • When this scheduled nightly task executes, does it do what it’s supposed to do?

The hypotheticals are difficult to spend time on because we, as testers, feel like we’re not getting credit for them.  We can’t prevent the team from having deployment problems.  But maybe we can ask enough questions to prepare them for the bad ones.


  1. fadzlizuka said...

    Ask enough questions to understand how the system config different better. Experienced the bad deployment and learn how they fix it. Overtime, from experience - you will see most of the problems is more or less having the same or repeatable every release (sigh) symptoms and can be troubleshoot the same way.
    Tester might not be able to solve it, but we can help to shed them some light during this rush time.

  2. Anonymous said...

    Don't forget the importance of a strong backout plan. A failed/delayed production deployment is bad enough, but one that you cannot recover from is much worse.

  3. Anonymous said...

    - Static Reviews of configs and production build package
    - Periodic audits comparing test and production environments (server settings, versions, service packs, load balancer configs, etc)
    - Deploy to DR or secondary production environment

  4. Kumar, Sanjay said...

    This is very common in these days, specially for the SaaS based organization. To overcome this issue,

    1. Testing team should well versed with configuration and deployment steps.

    2. Let the testing team also work with configuration managment team, when exactly the release / patch is getting deployed on production environment.

    3. Once the product / release gets rolled out to production, perform some smoke test and if you have your automated suit, run the same.

    I think, this is the right solution to overcome this issue.

    - Sanjay

Copyright 2006| Blogger Templates by GeckoandFly modified and converted to Blogger Beta by Blogcrowds.
No part of the content or the blog may be reproduced without prior written permission.