I attended Robert Sabourin’s “To Infinity And Beyond” class at STPCon. This guy’s sessions are fun because of his sense of humor and pop culture references. He played us a screen captured video of someone sending an email and we were asked to write down as many boundaries as we could think of within the video. I had nearly 40 in five minutes but many testers got more.

The point was to start a conversation about boundary tests that go beyond the obvious. The first take-away I embraced was a simple mnemonic Robert gave us for thinking about boundaries. I usually hate mnemonics because most of them are only useful as PowerPoint slide filler. However, I’ve actually started using this one:

“Take AIM”

A – Application logic
I – Inputs
M – Memory storage

These are three ways to think about boundaries for test ideas. Most features you attempt to test will have these types of boundaries. The other cool thing about this mnemonic is it prioritizes your tests (i.e., test the application logic first, if you have it).

The typical example is testing a text box. But let's apply it to something different. If we apply Robert’s mnemonic to the spec, “The system must allow the user to drop up to four selected XYZ Objects on a target screen area at the same time”, we may get the following boundary tests.

Application logic (i.e., business rules)

• Drop one XYZ object.
• Drop four XYZ objects.
• Attempt to drop five XYZ objects.

Inputs

• Attempt to drop one ABC object.
• Attempt to drop a selection set of three XYZ objects and one ABC object.
• Can the objects get to the target screen area without being dropped? If so, try it.

Memory storage

• Attempt to drop 1,000,000 XYZ objects (there may be a memory constraint in just evaluating the objects)
• Refresh the screen. Are the four dropped XYZ objects still in the target screen area?
• Reopen the AUT. Are the four dropped XYZ objects still in the target screen area?

What other boundary tests can you think of and which AIM category do they fit into?

Sorry about the blog lapse. I just got back from STPCon and a 10-day vacation in the Pacific Northwest.

I’ll share my STPCon ideas and takeaways in future posts. But first I have to mention my STPCon highlight.

During the conference, keynote speaker James Bach invited me to participate in an experiment to see how quickly testers can learn testing techniques. He approached me before lunch and I joined him and another tester, Michael Richardson, in the hotel lobby. James asked us each to play a different game with him (simultaneously). Michael played the “Dice Game”, in which Michael had to roll random dice of Michael’s choosing, from several different dice styles, to determine a pattern that James made up.

I played a game much like “20 Questions”, in which I had to explain what happened based on vague information I was given. I could ask as many Yes/No questions as I wanted. During the games, James coached us on different techniques. After about 45 minutes, Michael jumped in and bailed me out. Afterwards, James asked me to play again with a different premise. I was able to crack the second premise in less than 5 minutes. I would like to think I actually learned something from my first, painful, attempt.

These games share similarities to software testing because they require skillful elimination of infinite variables. Some of the techniques James suggested were:

• Focus then defocus. Sometimes you should target a specific detail. Other times, you should step back and think less narrow. For me, defocus was the most important approach.
• Forward then backward. Search for evidence to backup a theory. Then search for a theory based on evidence you have determined.
• Dumb questions can lead to interesting answers. Don’t be afraid to ask as many questions as you can, even if they are seemingly stupid questions.
• Flat lining. If you are getting nowhere with a technique, it’s time to stop. Try a different type of testing.

Later, James asked Michael to think up a challenging dice game pattern. James was able to solve it in about 30 seconds using only one die. Obviously, having played before, he knew it would eliminate most of the variables. I just used this exact idea today to understand a software behavior problem I initially thought complex.

After the stress of performing these challenges with James, we sat back and enjoyed a conversation I will not forget. James was more personable and less judgemental than I expected. Despite being a high school dropout, he is currently writing a book about learning (not about testing). I also thought it cool that during STPCon he headed across the street to the Boston Public Library and began reading from various history books. He was trying to determine why people with power would ever choose to give it up. I guess he’s got more on his mind than just testing.

For me, spending time with James was a huge privilege and I was grateful for the opportunity to discuss testing with him, as well as getting to know the person behind the celebrity.

Note: Michael Richardson was also an interesting guy. He once did QA for a fish company and was literally smacked in the face with a large fish due to poor testing.

Verifying fixed bugs logged by you is easy. You’re a tester. You logged a proper bug, with repro steps, expected results, and actual results. You remember the issue.

But what about those bugs logged by non-testers that you never even had a chance to experience? You certainly need to verify their fix. However, how can you be sure you understand the bug?

Here is what I try to do.

First, I track down to person who logged the bug. I ask them how it was discovered. Knowing this info will solidify the repro steps (if there were any).

Second, I experience the bug in its unfixed build. This sometimes means preventing the fix from being released to the environment I need to verify it on (so I have time to repro it). If I experience the problem I am usually confident I can verify its fix.

Third, I determine what was done to fix the bug? I can usually identify the dev from the bug history. Even if the bug has the dev comments about its fix, devs are always thrilled to discuss it at length.

Finally, while talking to the dev, I squeeze in the important question, “What related functionality is at risk of breaking?.” The answer to this question helps narrow down my regression test goal to something realistic.

Am I missing anything?

While cleaning my desk, I discovered a brand new promotional Mercury cap. Since I’m not a big fan of Mercury (e.g., TestDirector, QuickTestPro), I decided to award the cap to the tester on my team who could log the most bugs in a given week.

Life would be so much easier if our main goal were simply to log as many bugs as possible. Most of you will agree with me, that given enough time, you can find bugs in any piece of software thrown at you. Sure, they may be stupid bugs that nobody cares about, but you’ll find them.

Let’s face it. Testers don’t have very concrete goals. “Ensure the quality of the software”. Sometimes it’s fun to get a little competitive with each other and just focus on primitive stuff, like flooding your devs with bugs!

I think they should.

The less selfish you are about your bug list, the more bugs you’ll get logged, and the more info you’ll have about the state of the AUT. In my team, the devs, and the business people who write the requirements, all log bugs. This is an Agile team so we have the luxury of continuous business testing.

How are their bug reports? Terrible. The dev’s bug reports are usually too low level for me to understand. The business people’s bug reports usually lack repro steps, include multiple problems, and rarely describe what they expected. That being said, I wouldn’t trade them for anything.

As poorly as said bug reports may be written, they are tremendously valuable to me. Although a brief face-to-face is sometimes necessary to decipher the bug description, some of these bugs identify serious problems I have missed. Others give me ideas for tests (from a dev perspective or a business perspective). And it’s more convenient to retrieve these from the Bug list rather than rooting through email. Finally, I think most team members enjoy the ability to log bugs and even consider it a privilege.

I have some non-testers on my team who have logged more bugs than some of the testers on my team. Scary, huh? The point is, if a non-tester wants to log bugs, don’t feel threatened by it. Embrace it!

I’ll talk more specifically about how I handle bugs written by non-testers in a future post.

I sent an email to a dev today, asking why his new sort-by-column functionality for a grid was not actually sorting the values properly. For example: an ascending sort would rearrage alphanumeric data to something like "A", "B", "C", "A", "S", B".

Shortly thereafter the dev stopped by my desk and explained, "It sorts. It’s just not perfect."

I know it looks better on tester resumes to emphasize one’s White Box Testing abilities and brag about how many bugs you caught before they manifested on the UI. It also serves for far more condescending trash talk amongst testers. But since the majority of the testing I do is manual Black Box Testing, I often feel depressed, wondering if I am inferior to my testing peers and fellow bloggers.

The other day something occurred to me… Black Box testing is actually more challenging than White Box Testing. That is, if it is good Black Box testing.

I’m testing a winform app, that at any given time, may have about 6 different panes or zones displaying. The bulk of the user paths require drag/drop between various zones into grids. The possible inputs are nightmarish compared to those of the underlying services. Determining bug repro steps takes creativity, patience, and lots of coffee. Communicating those repro steps to others takes solid writing skills or in-person demos. And predicting what a user may do is far more challenging than predicting how one service may call another service.

I’m not suggesting apps should be or can be tested entirely using a Black Box approach. But the fact is, no matter how much white box testing one does, the UI still needs to be tested from the user’s perspective.

So if you’re feeling threatened by all those smarty pants testers writing unit tests and looking down on the black box testers, don’t. Effective Black Box Testing is a highly skilled job and you should be proud of your testing abilities!