As I struggle to find a tool that is current enough to automate tests for a Silverlight 4.0 app, my former QA manager and thoughtful testing buddy, keeps spouting the same message to me. Alex Kell keeps saying my first mistake was to let the programmers pick SL4 in the first place. We have tools that can automate SL2 and even SL3. Alex says, why not convince the programmers to use the older versions of SL.

His point being, if testing is really so important, then why shouldn’t we select our product’s coding technologies based on the ease of testing them? It’s that thing we always hear about called, “testability”.

When the initial SL4 decision was made, I sheepishly took Alex’s advice. I asked the lead architect if he could use SL3 instead. I explained how easy it would be to convert my existing automated test framework to drive SL3. He paused for two seconds, then said “no”, the SL4 bindings were superior and more favorable for our project. Can you blame him? I can’t. Of course, any team starting a greenfield project would want to begin with the latest supported code version.

So now I'm getting busy rewriting my framework using Microsoft’s CodedUI platform, which has SL4 support and is used elsewhere on my team.

In the meantime I’m haunted by Alex’s advice and wonder how hard I should have pushed. On the other hand, there is merit in adaptability. To be able to support your team with the best testing possible, based on their design choices…

…well that is something too.


What do you think?

Can you test something too much? Sure.

If you spend all your time testing FeatureA and four other Features go untested, you’ve probably tested FeatureA too much.

Part of what makes some testers better than others is their ability to know how much testing effort to put into each Feature and when. This is a tricky decision and may require the tester to stop testing FeatureA, even though FeatureA is still yielding bugs. What?!! Think about it. If you know FeatureA is yielding bugs but you don’t know anything about FeatureB, where should you spend your time next? I say, FeatureB. It just feels wrong, doesn’t it?

A lot of testers have a hard time with this. They get wrapped up in completing exhaustive testing on one Feature while the unknowns sit and collect dust. They follow their nose for the reward of bug discovery in comfortable areas. They underestimate how much time it will take to test the unknowns and before they know it, opps, not enough time to test everything!

Have you ever seen one of those plate spinning circus acts? You know the ones, the guy is frantically running around trying to keep all the plates spinning on sticks so they don’t fall. As a tester, you should be like a plate spinner. The Features are the plates. I don’t know what the sticks are, don’t worry about it. But if your programmers are working on Features (e.g., fixing bugs, finding bugs, refactoring, testing), your plates are spinning.

Keep your plates spinning!

At Monday’s retrospective, my answer to the question, “What should we do differently”, was to "have more fun". We have been cranking out releases for 85 iterations and the new year seemed like a good time to try something fresh. One of my testers came up with Bug Bucks.

Although it violates several agile practices, we’re going to give it a try. Here’s how it works:
  • Each programmer earns Buck Bucks equal to the magnitude of Features they code (e.g., a Feature with a magnitude (AKA complexity) of 13 would earn the programmer 13 Bug Bucks).
  • Any tester, BA, or other programmer who finds a bug in said Feature, gets to take away a Bug Buck from the programmer who coded it. The programmer only loses the Bug Buck if the team decides to fix it.
  • Bugs found in production subtract a Bug Buck from each team member (all testers, BAs, and programmers).
  • We have various denominations of Bug Bucks to make change, and team members will pin them to their cubes as they acquire them.
So what can one do with a Bug Buck? We’re still working on that but pending budget approval, we will let them be exchanged for gifts, or more likely, one minute of personal time off, which should work out to a free day off per quarter for ambitious players!

There are several problems we are aware of; such as, will this discourage collaboration? Um…yes.

But will it really? In the big picture? We've already been collaborating on the rules. Anyway, our current approach is not to over-engineer this game, but to try to have some fun. If our velocity changes for the worst, we’ll pull the plug... just have to wait and see.

I'll report back.

Song to the famous Christmas Song, "Let it Snow, Let it Snow, Let it Snow"

The performance test stats are frightful,
Useability is not delightful,
I expect the bug count to grow,
App is slow! App is slow! App is slow!

It doesn't show signs of stopping,
And this build is really flopping,
My confidence is way down low,
App is slow! App is slow! App is slow!

When the hour glass goes away,
And the screen finally starts to repaint,
There's a timeout error trying to say,
Ready for prod, this thing ain't!

Available memory is slowly dying,
But, my devs, are still denying,
They say it's my box, but I know,
App is slow! App is slow! App is slow!

Last year's Twas the Night Before Prod Release was much better.


Last week a group of us testers, programmers, and business analysts volunteered to sort, inspect, and pack donated food, beverages and health products at the Atlanta Community Food Bank. Most cities have similar gigs and this is a fun way of doing volunteer work that is related to the old school usage of the term QA. If you’re a tester, volunteer to be an inspector.
As an inspector, you’ll go through all the donated items and decide which get packaged up to become meals, which get thrown in the trash, or which need special treatment. This is QA at its most primitive and I found it incredibly fun because unlike software testing, I actually got to make my own decisions as far as what would go to production and what would not.
  • Black Box Testing - Regular food (e.g., canned soup, cereal, pasta) would get thrown away if its expiration date was <>
  • White Box Testing – Sometimes food looks good on the outside, but if you screw off the peanut butter jar lid and look inside, you may find someone has broken the inner seal and helped themselves to a few spoonfuls.
  • Cosmetic Testing – Cans with no labels or bags of food without an ingredient listing were rejected.
  • Bugs – Literally. Reject it if the food was exposed. Finding damaged secondary packaging (e.g., cardboard cereal box ripped but plastic bag intact) was like finding a software bug. Let the dev tape it back together, then you can accept it.
  • Usability Testing – Some cans are smashed; can they easily be opened? Is it so smashed the user can injure themselves on sharp corners?
  • Security Testing – Was the product recalled? Example: Campbell’s Spaghettios were recalled due to uncooked meatballs. Fail!
Things that helped us succeed:
  • Motivation – we were told what the average work performed by volunteers was. We consider ourselves above average so we set a goal to beat the norm. Thus, we worked hard to achieve our goal and prove we were the best.
  • Roles – We were assigned to roles as either inspector, sorter, or packer. When any role was caught up, we switched roles to keep busy and keep things moving.
  • Collaboration – “I forgot, what do we do with baby food?”, “What is the rule for powdered milk?”, “Can you hold this shut while I tape it up?”. We worked side-by-side in the same room and answered each other’s questions.
  • Breaks – They forced us to take breaks. Without the breaks, fatigue may have set in. The breaks allowed us to reflect and exchange stories about mistakes we had made (spilling salt all over the place) or gross encounters (sticky stuff leaking out of jar). This also motivated us to work harder after the break, attempting to encounter good story material. They gave us free beverages and chocolate during the break (donated chocolate can’t get packaged because it melts), which means we didn’t have to be distracted with food/beverage as we worked.
  • Oracles – When we couldn’t help ourselves, we knew which volunteers were the oracles. They knew all the answers based on prior experience.
  • Music – Who doesn’t love listening to Michael Jackson’s Thriller?
After a mere two hours of work, we beat the average by packaging 8,109 lbs of food, which translates into 5,460 meals.

...if only we could weed through software bits this quickly.

Richard Siemens responded to my Testers, Stay Frosty. Fight Tester Fatigue post and asked if I have any ideas on how to combat tester fatigue. I’m glad you asked, Richard, because my previous post was too lame to suggest any.

Here is what I think:

Stuff the tester can do to combat test fatigue:
  • Treat yourself to static goals on occasion. For example, if you say “my goal today is to verify all the fixed bugs currently on this list”, it has a firm stopping point; the current list does not include the bugs that will be on that list in an hour. However, if you say “my goal today is to verify all the fixed bugs”, I’ll bet the devs are cranking out the fixes just as quickly as you verify them and the list just won’t clear. How unsatisfying. It’s like walking up the down escalator.
  • Use an approach similar to Session Based Test Management where you follow a time-blocked mission. As you encounter bugs that feel annoying because they take you off track, just ignore them! That’s right…at least ignore them for now. Make a quick note, then go back and investigate said bugs when you and your team decide it is time.
  • Show your manager a list of all the stuff you need to do and ask them to prioritize it. You may be surprised at some of the stuff with a low priority. In fact, if you make that list long enough, you’ll probably get some stuff knocked off it.
  • Stop Testing and do something else. Take time each week to improve your testing skills. And don't tell me you're too busy. I don't buy it. It’s those testers who get complacent and test the same way day-after-day that make us look bad. Take a few hours each week to read testing blogs, start your own blog, write a program, improve your typing skills, read a chapter from a testing book, or just hang out in the break room chatting it up with your support team. Any decent manager will appreciate a happy employee taking a break to become a better tester in some way. Because any decent manager knows a better tester gets more work done when they do test.

Stuff the test manager can do to combat test fatigue:
  • Be more careful about what you reward. Instead of placing so much weight on completing test work on time. Go out of your way to reward discovered bugs, even right before deadlines; “Nice bug catch, that was close!”.
  • Assign non-testing tasks to testers. As testers, it’s refreshing to work on non-testing tasks every so often because testers can control non-testing tasks. These have hard and fast completion criteria and you only have to worry about doing it one way. Examples include:

    Organizing regression tests
    Building test metric report
    Giving tester presentations
    Organizing a team outing
    Facilitating a retrospective
    Updating test environments
    Engineering process improvements
  • Let the whole development team know how much work the testers are doing. Let the testers know too. An easy way to do this is nightly team test status emails.
  • Encourage a rest period during test cycles.

    Every morning, before work, I put in 40 minutes on my YMCA’s StairMaster, whilst reading my Economist. I use the “Interval” workout setting, which requires two input parameters; Workout Level & Rest Level. Over the last two years, I’ve steadily been increasing the Workout Level, while keeping the Rest Level about the same. When the machine kicks in to the Workout Level, I give it everything I’ve got. Three minutes later, when I’m about to give up, it rewards me with the Rest Level and man does it feel good. Then the cycle repeats.

    I think testers should follow a model similar to my StairMaster’s Interval workout. My teams have 4 week iterations. I stay out of everybody’s hair during week 1 and try to set the mood that it's a rest week. We still attempt to test as early as possible, we just don’t crank up the intensity until about week 2 and 3. We finish all new feature testing by end of week three, then begin to come back down during week 4 by concentrating on regression testing.

    Just like my workouts, as we grow our tester skills, product knowledge, and team cadence, we should be able to increase our test level with each iteration, while allowing ourselves to maintain the same rest level.

Based on my own test experiences and those of my testers, I've noticed the following.

At the start of a test cycle, if your test fails, your likely reaction is:
“Yeah, baby, it failed! Yesssss! I rock!”

At the end of a test cycle, if your test fails, your likely reaction approaches:
“Damn! I can’t believe it failed. We’re never going to get this done in time. ...On second thought, maybe it didn’t actually fail. Maybe I did something wrong, I heard the DBA was doing some kind of maintenance, maybe that was the problem. Besides, the production servers are much faster, I’m sure they would work better. Perhaps if I reboot and try again, it will work. Then I won't have to tell anybody.”

Can you relate on some level? Finding bugs in fresh software gives us a rush. We joke about it; “Let me sink my teeth into your code!”. But after a while, we get tired of finding bugs. We just want stuff to work so we can move on. As we approach the ship date, we start to feel frustrated when stuff crashes. We’re actually…wait for it…disappointed to find another bug. We wish the test had passed.

Testers, be careful. Don’t ever let yourself grow tired of finding problems. When that happens, your ability to investigate diminishes and your team value drops. I call this “Tester Fatigue”. Being aware of tester fatigue is probably all you need to know to avoid it and stay frosty.



Copyright 2006| Blogger Templates by GeckoandFly modified and converted to Blogger Beta by Blogcrowds.
No part of the content or the blog may be reproduced without prior written permission.