When I first began getting paid to test software, I was a little confused. I knew it was invigorating to catch scary bugs prior to production but I wasn't really sure how valuable my job was to my dev team or the rest of the world! In fact, I didn't really know if testing software was anything to make a career out of in the first place.
A few years ago I came across Harry Robinson's Bumper Stickers for Testers post on StickyMinds.com. It was at that point that I decided my job as a software tester was valuable (and even a little cool). Harry and all the other software testers who contributed the excellent material on said post inspired me to take pride in my job and now I even sport a couple bumper stickers to show it (see below). If you test software, I encourage you to do the same.
Allen, one of the developers on my team, recently told me about some poorly written bugs he had received from a tester. Allen said,
"Basically these bugs describe the application behavior as a result of user actions working with an application, without any explicit identification of the buggy behavior. This tester wrote many, many bugs with one or two sentences. His bug description pattern was like this:
When I do A, it does B.
Examples:
- When I try to move the video object, the background image stays still. (OK. Where is the bug?)
- When I set the video file to a path located in another course, the file is copied to the current course when rendering the page in IE browser. (To some, this may be the correct behavior. So where is the bug?)
This is an ambiguous way to describe a bug. It frustrates me! "
About a year ago, Allen offered similar candor on several of my bugs. Since then, I have made it a point to force the following two statements into every bug I log.
Expected Results:
Actual Results:
No, it’s not an original idea. But it is an excellent idea that makes it nearly impossible for one to ever log another ambiguous bug. If the tester had used it for one of Allen’s examples above, we might see the following.
Expected Results: The background image moves with the video object.
Actual Results: The background image does not move with the video object. Instead, the background image stays still.
Expected Results/Actual Results. Don't log a bug without these!
I have a recurring dream when I get sick. In the dream I’m tasked with riding an elevator up and down to various floors. On each floor I encounter a bunch of numbers bouncing around acting out of control. Part of my task is to make sense out of these numbers on each floor. Like, maybe put them in order…I’m never sure. However, I usually give up and ride the elevator to another random floor and try again. The dream just loops through this sequence. It sounds silly but during the dream it is quite scary and unstoppable for some reason.
What does this have to do with testing? The state of our project feels so chaotic that I keep getting flashes of this dream. My days are filled getting emails or people stopping by my cube with error messages accompanied with vague or no repro steps. Each email is critical and the next seems to preempt the previous. The amount of tests I’ve actually executed myself has been minimal over the past month. Instead, I’ve been attempting to determine what other people are doing. The email threads typically get hijacked by people chunking in non-related problems and I can often identify people misunderstanding each other because each thread becomes more ambiguous than the next. These threads contain bugs that get lost because nobody can figure out enough info to log them. Ahhhhhhh! So I get back on the elevator and see if I can make sense out of the next floor.
IMHO, much of this chaos could be avoided if people would log the bug, no matter how few infos are known. In an extreme case, I still believe it would be valuable to log the bug if all you have is a crumby screen capture of an error. Something like:
Bug#20074 - “No repro steps but someone got this error once…”
The next time this error is encountered we now have something to compare it with. “Hey, this is the same error as Bug#20074, did we notice any clues this time? No?” Well, we can at least update the bug to indicate we saw the error again in the next build and someone else got it.” The emails referring to this problem can say “This may be Bug#20074”. And so on. Once we have a bug, no matter how hollow the bug is, the problem becomes more than someone’s sighting of Bigfoot. It becomes a problem we can actually collect information against in an organized manner. And hopefully, I can stop riding the elevator.
I recently began Integration Testing between two AUTs. Each AUT has its own bug DB and dev team. Let’s say we execute a simple test like this...
Step 1: Trigger EventA in AUT1.
Expected Result: EventB occurs in AUT2.
Actual Result: EventB does not occur in AUT2.
We’ve got the repro steps. Can we log it as a bug? Not yet, because where do we log it? AUT1 or AUT2’s bug DB? In this type of testing, I think fewer bugs are logged per hour. I believe this is because the burden on the tester to perform a deep investigation is much higher than during Integration Testing between different modules of the same AUT.
Part of the problem may be a pride-thing between the dev teams. Each dev team believes the bug belongs to the other dev team until proven wrong by the tester. Yikes...not very healthy! This dev team pride-thing may exist because devs are not always fond of collaborating with other dev teams. On my floor, the QA teams are forced to work together while the dev teams somehow manage to survive in their own clans. There are exceptions, of course.
Anyway, after adequate bug research, it is usually possible to determine bug ownership. But what if neither dev team is at fault? What if the bug exists merely due to an integration design oversight? Where do you log it then?
Boy, being a software tester is hard work!
When one of my devs asked me to repro a bug on his box because he didn't know how to execute my repro steps, I didn't think twice. But I was a little surprised when he stated his position that he was strongly in favor of QA always performing the repro steps on the developer's box instead of the developer doing it themself. He argued the devs didn't always have time to go through time consuming or complex repro steps. He attempted to retract portions of his statement once he found out I was blogging it...too late.
I should, however, add that this particular dev has been tremendously helpful to me in setting up tests and helping me understand their results.
That being said, I've never heard anything like this from a developer in all my years of testing and I think it's ridiculous. But it did make me think about how we speak to our devs via bug reports. When we log repro steps from blackbox tests we use a kind of domain specific language that requires a user-based understanding of the AUT. To perform a repro step like "Build an alternate invoice", one must understand all the micro steps required to build the alternate invoice and one may have to understand what an alternate invoice is in the first place. If the next repro step is "Release the invoice to SystemX", one must know how to release the invoice, etc.
I think it is realistic to expect developers to have an understanding of this business-centric language as well as knowing how to perform said procedures from the AUT. And in general, time spent learning and using the AUT will help the developers improve it's overall quality.
Am I right?
That is the question. Or at least, hypothetically, it could be. As my current project nears its "Go-No-Go" (I really hate that phrase) date, my decisions on how I spend my dwindling time are becoming as critical as the bugs I still have to find.
My former manager, Alex, and I had an interesting disagreement today. If given a choice between verifying fixed bugs or searching for new ones, which would be more valuable to the project at the bitter end of its development life? Alex said verifying fixed bugs because you know the code around those bugs has been fiddled with and the chances of related bugs are significant. I would instead choose to spend this time searching for new bugs because I believe these unexecuted tests are more dangerous than those bugs said to have been fixed.
Well, it all boils down to a bunch of variables I guess...
- How critical are these unexecuted tests or do we even know what they are?
- What does history tell us about the percentage of our fixed bugs that get reopened after retest?
- How critical are said fixed bugs?
The main reason we entered this discussion in the first place was because I am stubborn when it comes to fixed bug retesting (we call it "verifying bug fixes"). I find it dull and a waste of my skills. It seems more like busy work. The test steps are the repro steps and the outcome is typically boring. "Yay! It's fixed!" is less interesting than "Yay! I can't wait to log this!".
What do you think?
The first software test blogger I read was The Braidy Tester. He is still my favorite.
I borrowed from his test automation framework, took Michael Bolton's Rapid Software Testing course based on his suggestion, and laugh at his testing song satires. But most of all, The Braidy Tester (AKA "Micahel" or "Michael Hunter") inspired me to think more about testing and how to improve it.
So when he asked to interview me for his Book of Testing series I was thrilled. I realize this post is nothing more than an attempt to rub my own ego, but perhaps my answers to the questions will help you think about your own answers. Here it is...
http://www.ddj.com/blog/debugblog/archives/2008/01/five_questions_43.html
RSS
