App is Slow! App is Slow! App is Slow!
2 comments Posted by Eric Jacobson at Wednesday, December 22, 2010Song to the famous Christmas Song, "Let it Snow, Let it Snow, Let it Snow"
The performance test stats are frightful,
Useability is not delightful,
I expect the bug count to grow,
App is slow! App is slow! App is slow!
It doesn't show signs of stopping,
And this build is really flopping,
My confidence is way down low,
App is slow! App is slow! App is slow!
When the hour glass goes away,
And the screen finally starts to repaint,
There's a timeout error trying to say,
Ready for prod, this thing ain't!
Available memory is slowly dying,
But, my devs, are still denying,
They say it's my box, but I know,
App is slow! App is slow! App is slow!
Last year's Twas the Night Before Prod Release was much better.
Last week a group of us testers, programmers, and business analysts volunteered to sort, inspect, and pack donated food, beverages and health products at the Atlanta Community Food Bank. Most cities have similar gigs and this is a fun way of doing volunteer work that is related to the old school usage of the term QA. If you’re a tester, volunteer to be an inspector.
As an inspector, you’ll go through all the donated items and decide which get packaged up to become meals, which get thrown in the trash, or which need special treatment. This is QA at its most primitive and I found it incredibly fun because unlike software testing, I actually got to make my own decisions as far as what would go to production and what would not.
- Black Box Testing - Regular food (e.g., canned soup, cereal, pasta) would get thrown away if its expiration date was <>
- White Box Testing – Sometimes food looks good on the outside, but if you screw off the peanut butter jar lid and look inside, you may find someone has broken the inner seal and helped themselves to a few spoonfuls.
- Cosmetic Testing – Cans with no labels or bags of food without an ingredient listing were rejected.
- Bugs – Literally. Reject it if the food was exposed. Finding damaged secondary packaging (e.g., cardboard cereal box ripped but plastic bag intact) was like finding a software bug. Let the dev tape it back together, then you can accept it.
- Usability Testing – Some cans are smashed; can they easily be opened? Is it so smashed the user can injure themselves on sharp corners?
- Security Testing – Was the product recalled? Example: Campbell’s Spaghettios were recalled due to uncooked meatballs. Fail!
Things that helped us succeed:
- Motivation – we were told what the average work performed by volunteers was. We consider ourselves above average so we set a goal to beat the norm. Thus, we worked hard to achieve our goal and prove we were the best.
- Roles – We were assigned to roles as either inspector, sorter, or packer. When any role was caught up, we switched roles to keep busy and keep things moving.
- Collaboration – “I forgot, what do we do with baby food?”, “What is the rule for powdered milk?”, “Can you hold this shut while I tape it up?”. We worked side-by-side in the same room and answered each other’s questions.
- Breaks – They forced us to take breaks. Without the breaks, fatigue may have set in. The breaks allowed us to reflect and exchange stories about mistakes we had made (spilling salt all over the place) or gross encounters (sticky stuff leaking out of jar). This also motivated us to work harder after the break, attempting to encounter good story material. They gave us free beverages and chocolate during the break (donated chocolate can’t get packaged because it melts), which means we didn’t have to be distracted with food/beverage as we worked.
- Oracles – When we couldn’t help ourselves, we knew which volunteers were the oracles. They knew all the answers based on prior experience.
- Music – Who doesn’t love listening to Michael Jackson’s Thriller?
After a mere two hours of work, we beat the average by packaging 8,109 lbs of food, which translates into 5,460 meals.
...if only we could weed through software bits this quickly.
Richard Siemens responded to my Testers, Stay Frosty. Fight Tester Fatigue post and asked if I have any ideas on how to combat tester fatigue. I’m glad you asked, Richard, because my previous post was too lame to suggest any.
Here is what I think:
Stuff the tester can do to combat test fatigue:
- Treat yourself to static goals on occasion. For example, if you say “my goal today is to verify all the fixed bugs currently on this list”, it has a firm stopping point; the current list does not include the bugs that will be on that list in an hour. However, if you say “my goal today is to verify all the fixed bugs”, I’ll bet the devs are cranking out the fixes just as quickly as you verify them and the list just won’t clear. How unsatisfying. It’s like walking up the down escalator.
- Use an approach similar to Session Based Test Management where you follow a time-blocked mission. As you encounter bugs that feel annoying because they take you off track, just ignore them! That’s right…at least ignore them for now. Make a quick note, then go back and investigate said bugs when you and your team decide it is time.
- Show your manager a list of all the stuff you need to do and ask them to prioritize it. You may be surprised at some of the stuff with a low priority. In fact, if you make that list long enough, you’ll probably get some stuff knocked off it.
- Stop Testing and do something else. Take time each week to improve your testing skills. And don't tell me you're too busy. I don't buy it. It’s those testers who get complacent and test the same way day-after-day that make us look bad. Take a few hours each week to read testing blogs, start your own blog, write a program, improve your typing skills, read a chapter from a testing book, or just hang out in the break room chatting it up with your support team. Any decent manager will appreciate a happy employee taking a break to become a better tester in some way. Because any decent manager knows a better tester gets more work done when they do test.
Stuff the test manager can do to combat test fatigue:
- Be more careful about what you reward. Instead of placing so much weight on completing test work on time. Go out of your way to reward discovered bugs, even right before deadlines; “Nice bug catch, that was close!”.
- Assign non-testing tasks to testers. As testers, it’s refreshing to work on non-testing tasks every so often because testers can control non-testing tasks. These have hard and fast completion criteria and you only have to worry about doing it one way. Examples include:
Organizing regression tests
Building test metric report
Giving tester presentations
Organizing a team outing
Facilitating a retrospective
Updating test environments
Engineering process improvements
- Let the whole development team know how much work the testers are doing. Let the testers know too. An easy way to do this is nightly team test status emails.
- Encourage a rest period during test cycles.
Every morning, before work, I put in 40 minutes on my YMCA’s StairMaster, whilst reading my Economist. I use the “Interval” workout setting, which requires two input parameters; Workout Level & Rest Level. Over the last two years, I’ve steadily been increasing the Workout Level, while keeping the Rest Level about the same. When the machine kicks in to the Workout Level, I give it everything I’ve got. Three minutes later, when I’m about to give up, it rewards me with the Rest Level and man does it feel good. Then the cycle repeats.
I think testers should follow a model similar to my StairMaster’s Interval workout. My teams have 4 week iterations. I stay out of everybody’s hair during week 1 and try to set the mood that it's a rest week. We still attempt to test as early as possible, we just don’t crank up the intensity until about week 2 and 3. We finish all new feature testing by end of week three, then begin to come back down during week 4 by concentrating on regression testing.
Just like my workouts, as we grow our tester skills, product knowledge, and team cadence, we should be able to increase our test level with each iteration, while allowing ourselves to maintain the same rest level.
Testers, Stay Frosty. Fight Tester Fatigue
4 comments Posted by Eric Jacobson at Thursday, November 11, 2010Based on my own test experiences and those of my testers, I've noticed the following.
At the start of a test cycle, if your test fails, your likely reaction is:
“Yeah, baby, it failed! Yesssss! I rock!”
At the end of a test cycle, if your test fails, your likely reaction approaches:
“Damn! I can’t believe it failed. We’re never going to get this done in time. ...On second thought, maybe it didn’t actually fail. Maybe I did something wrong, I heard the DBA was doing some kind of maintenance, maybe that was the problem. Besides, the production servers are much faster, I’m sure they would work better. Perhaps if I reboot and try again, it will work. Then I won't have to tell anybody.”
Can you relate on some level? Finding bugs in fresh software gives us a rush. We joke about it; “Let me sink my teeth into your code!”. But after a while, we get tired of finding bugs. We just want stuff to work so we can move on. As we approach the ship date, we start to feel frustrated when stuff crashes. We’re actually…wait for it…disappointed to find another bug. We wish the test had passed.
Testers, be careful. Don’t ever let yourself grow tired of finding problems. When that happens, your ability to investigate diminishes and your team value drops. I call this “Tester Fatigue”. Being aware of tester fatigue is probably all you need to know to avoid it and stay frosty.
The Better You Test, The Worse You Look
1 comments Posted by Eric Jacobson at Friday, November 05, 2010It’s just not fair.
The better we test, the more we appear to not meet our deadlines.
Skilled testers provide more feedback than unskilled testers. The skilled testers find more bugs and raise more questions. The more bugs found, the more testing is required to verify the bugs. The more bugs that are fixed, the more testing is required to regression test.
The unskilled tester scratches the surface. If no bugs or questions are discovered and little feedback (e.g., test results) is produced, the unskilled tester calls it a day at 5PM everyday and naively goes home to watch TV. It’s possible to get away with this, especially when the missed defects are never discovered in production, and those that are, may be written off as too difficult to catch in test. Poor performers can hide well in the test world. You may know some.
What can we do about this frustrating injustice?
Reduce feature ownership.
The above paradigm may be partly the result of feature ownership. If the testers are each assigned certain features to test and therefore only responsible for seeing those features through to production, we see the unskilled tester rewarded with easily meeting deadlines, and the skilled tester pulling her hair out, trying to keep up.
Test managers have some control over this. They can ask the unskilled tester to assist the skilled tester with less cognitive tasks, such as bug verification or regression testing. This helps accentuate the team mentality, that nobody goes home until all features are fully tested. Most Agile teams are already doing this but I suspect the unskilled testers still manage to provide less value on Stories they pull from the task board.
Deadlines are not the main goal.
In most trades, we reward people for getting work done on time. Perhaps in testing, we should stop doing this. It’s almost as if we should do the opposite; reward testers for managing to keep the team busy fixing problems and thus not meeting the deadline.
I’m exaggerating, of course, but when we tell testers to “get this tested well and on time”, there is a conflict of interest. To make matters worse, it’s easier to look at a clock and say “great job, tester, you completed the testing on time” than it is to look at a piece of software and say “great job, tester, you tested this well”.
Let’s not forget to celebrate the efforts of those testers who always seem to be swamped and having a tough time meeting team test deadlines. They need a break sometimes too.
Don't Bother With Microsoft Test Manager 2010
74 comments Posted by Eric Jacobson at Wednesday, October 27, 2010After attempting to use Microsoft Test Manager 2010 for an iteration, we quickly decided not to use it. Here is why.
About 3 years ago we finally managed to stop using HP Quality Center (AKA Test Director). We started managing our test cases and bugs as work items in Microsoft Team Foundation Server (TFS). The brilliance behind this transition was that it gave us the ability to attach our tests and bugs to the iteration’s Features/Stories and Tasks. This meant, as Features move in and out of iterations, the related tests and bugs follow; a beautiful thing! Additional benefits are, 1.) Programmers and BAs can easily review our test cases and write reports based on their execution status and, 2.) no more attempting to synch the TFS Features to Quality Center Requirements…the horror! I totally hated that!
It turns out, Microsoft Test Manager 2010, which sits on Microsoft TFS, took away many of the above TFS benefits and added as much overhead as Quality Center.
Stuff We Didn’t Like About Microsoft Test Manager 2010:
- Before one can write tests, one must set up a Test Suite. Test Suites do not update when the Feature set changes. Thus, one must manually keep one’s Test Suite synched with one’s iteration. To further frustrate, without customization, Test Manager does not let one write test cases for Task work items.
- Test cases in Test Manager follow a different workflow than those of TFS. The result is, nobody on your team can see which of your tests pass or fail unless they open Test Manager, which they probably don’t have a license for (unless they are testers). The reasoning behind this is probably Test Manager’s test cases can have test case run execution history (e.g., TestA could be passed in one build and failed in another build at the same time ). Test case run execution history is actually cool. In fact, it was one of the biggest motivators for us to try Test Manager. However, we were hoping Test Manager would trickle down the results to TFS so the whole team could benefit.
- One of the most annoying parts of our Test Manager trial may have been its usability and performance. The screens frequently hung, causing some testers to force quit and re-launch. The navigation was also awkward. It took too many clicks to get anything done.
- To update the execution status of Test Manager tests, one must go through the ridiculous test case executor. This is the thing that shows you each test case step and asks you for a pass/fail result. I can’t imagine anyone actually testing like this. Quality Center had something similar but provided an alternate method of updating execution status (i.e., one could bulk update via a grid).
- Our other gripe about updating the execution status of Test Manager tests is that the test case “summary” does not show. Most testers like to write their test cases as fragments, using the test case work item’s free-form summary tab. The summary tab is preferred over the grid, which forces tests into individual steps with expected results. The big joke is, if you write your tests in the summary tab, it is not possible to see them while running the silly test case executor. So you are presented with a blank test step and asked if it passes or fails.
Stuff We Think We Like About Microsoft Test Manager 2010:
There are two things some project teams are planning on using Test Manager for in the near future:
- Calling our CodedUI test methods and passing in parameters. According the idealistic demo I saw at the Stareast Microsoft booth, this allows manual testers to write/execute automated tests without coding.
- Using test case run execution for regression testing.
Conclusion:
I guess TFS's simple work item model fits our needs for flexible lightweight test documentation with little administrative overhead. Maybe someone can convince me otherwise.
Many testers have chosen to make their jobs stressful by taking on more responsibilities than they should, obscuring their skills with those of others on their teams. Choosing to make your job less stressful will not only help you enjoy testing, it will also allow you to focus on testing, and improve your standing as a test leader. Here are some things I’ve learned over the years…
- When people ask me “Did you QA Certify this for production?”, I remind them the question of when to ship is a business decision, but I can tell them much of what they need to know (e.g., how it currently works under certain conditions, what bugs exist) to make that business decision.
- When I hear users complain about the product not working for them due to the way it was designed, I feel empathy for the users….then I remind myself that I never tell BAs/devs how to build the product or what it should do.
- When I’m faced with really scary technical things to test, I turn to my team. I only have to look stupid to the first person I talk to, because by the time I get to the second person, at least I have what I learned from the first. As I continue to share my test ideas, they gradually change from lame to sophisticated. Soon, I realize everybody else on my team was just as confused as I was.
- Crunch time. I expect it. I chillax at the beginning of an iteration and work harder/longer towards the end. I maintain my work/life balance by keeping my personal calendar free right before and after a production release. Working late with other team members is often just as fun as spending a quiet evening at home with my wife (don’t worry, she doesn’t read my blog).
- Too much to test! Too little time! This one still stresses me out on occasion. But when I’m thinking rationally, I pose the question to my BAs, “I have 2 days left, would you prefer I test these new features or focus on regression testing?”. It trains them to respect my schedule and understand that it is finite. It also shows that I respect their business sense and value their opinion.
- Your product went live and the quality sucks. Okay, you can feel somewhat guilty...along with your devs and BAs. But remember, you didn’t code or design it. Quality can only be added by the programmers (e.g., if you have no code, you have no quality.). If it sucks now, just think about how much it would have sucked before those 471 bugs you caught!
What things do you do to make testing less stressful and maintain your sanity?
Subscribe to:
Posts (Atom)
RSS
