Stressed Testing

Just before the holidays, we went live with another relatively huge chunk of users who require slightly different features than our previous users. The bug DB is quickly filling up with bugs discovered in production. These bugs are logged by the business/support arm of our team because testers can’t keep up. Many of the bugs don’t have repro steps and appear to be related to multiple users, performance, deadlocking, or misunderstood features. Other bugs are straight forward; oversights uncovered after users take the app through new paths for the first time.

My team is struggling to patch critical production issues to keep the users working through their deadlines. I want to investigate every new bug to determine the repro steps and prepare for verifying their fixes. Instead, I’m jumping from one patch to the next, attempting to certify the patches for production. Keeping up is difficult. New emails arrive every few seconds.

This is an awkward phase, but the team is reacting well, maintaining a good reputation for quick fixes. Nevertheless, I’m stressed.

Am I doing something wrong?
Do I suck for letting these bugs get to prod in the first place?
Should I be working late every night to clean up the bug DB?
Am I the bottleneck, too slow at getting patches to users?
Should I certify patches that are only partially fixed?
Should I be writing new tests to verify these bugs and prevent them from returning?
Should I be out in the trenches, watching the user behavior?

Can you relate?


  1. Michele Smith said...

    Am I doing something wrong?
    Is there a team involved in the effort? If so, you cannot really ask if you are doing something wrong, but what can you all do different, if anything, next time. Write down things that aren’t quite right in how you all are doing so that you won’t forget what can be improved next time. It is often a good thing to gather together, when things settle down a bit, and discuss what went right and what went wrong.

    Do I suck for letting these bugs get to prod in the first place?
    Why, did you distract the other testers? Did you find bugs and not report them? Were you sleeping? Or, did you do the best you could, with what you had, at that time?

    Am I the bottleneck, too slow at getting patches to users?
    Are you tying up management? Are you preventing them from releasing the patches? Are you making the business decisions as well as testing the application/product/system?

    Should I be out in the trenches, watching the user behavior?
    More bugs may escape and less patches certified if you are watching the users instead of testing, so probably not. But by reading the bug reports and communicating with the support people you may learn a lot about how the customers use/abuse the product/application/system.

    Can I relate?
    As a tester myself, I hate when the customers find a really good bug that we (the test team) missed. But I also know we can’t find all the bugs, test completely, or make the business decisions to release the products.
    When I feel like you describe in the post, I generally take a step back as soon as there is an opportunity to do so, and do something to relax. When I have my own sense of being back, I remember who I am (my values, ethics, and principles). When I remember these things I remember that the situation cannot beat me up, and I can approach it with my attitude in a better place and my head in a better frame of mind.

  2. Eric Jacobson said...

    Thanks, Michele! Your comments made me feel better.

    I think will ask for help compiling a list of emergency production patches and hold a little meeting to determine what the testers can do better next time.

    It will help ensure we are not just responding to reported issues, we are also adding some extra value to prevent history from repeating itself.

  3. Alex said...

    One thing important to remember is that you're not the cop. You're there to provide as much information about the software to your customers and management so that they can make the proper business decisions.
    Be clear about all the work you see out there that's not getting done, and what the risks are if you let that work hang around undone.

    And of course I can relate. I've been there! Literally!

Copyright 2006| Blogger Templates by GeckoandFly modified and converted to Blogger Beta by Blogcrowds.
No part of the content or the blog may be reproduced without prior written permission.