Should you log a bug without knowing the expected results?
Someone chuckled at my “?” for the expected results, during a bug review meeting. The bug (like many of my bugs) looked like this:
Repro Steps:
1. do something
2. do something else
3. do something else
Expected Results: ?
Actual Results: an unhandled error is thrown
Good testers determine scenarios that nobody thought of. That is one skill that makes us good testers. Some time ago, I didn’t let myself log bugs until I tracked down the proper oracles and determined the expected results; a practice that sounds good…until you try it. Unfortunately, the oracles are never quick to respond. Often they pose questions to the users, which open additional discussions, meetings, etc…until the tester forgets the details of their wonderful test.
So these days, if I encounter a bug and I don’t know the expected results, I log the bug immediately and let someone else worry about expected results…if they even care. It’s not because I’m too lazy to seek out my oracles. It’s because my time is better spent logging more bugs! When in doubt, remember, the tester’s primary job is to provide information about the AUT. It’s not the responsibility of the tester to determine expected results. If the tester identifies a scenario that will crash the AUT, they should log the bug now.
My opinions do not reflect those of my employer.
Subscribe to posts
RSS
Popular Posts
-
After attempting to use Microsoft Test Manager 2010 for an iteration, we quickly decided not to use it. Here is why. About 3 years ago we f...
-
Data warehouse (DW) testing is a far cry from functional testing. As testers, we need to let the team know if the DW dimension, fact, and b...
-
I recently read about 15 resumes for tester positions on my team. None of them told us anything about how well the candidate can test. Here...
-
Want your bug reports to be clear? Don’t tell us about the bug in the repro steps. If your bug reports include Repro Steps and Results se...
-
When someone walks up to your desk and asks, “How’s the testing going?”, a good answer depends on remembering to tell that person the right ...
Blog Archive
Labels
- Teamwork (86)
- bugs (81)
- process (66)
- software testing career (49)
- automation (45)
- writing tests (38)
- Personal Excellence (37)
- Managing Testing (33)
- questions (31)
- language (29)
- testing metaphor (23)
- Tools (19)
- STPCon (10)
- heuristics (10)
- Test Cases (9)
- test blogs (9)
- CAST (8)
- Presentations (8)
- Test This (8)
- metrics (8)
- Rapid Software Testing (7)
- Silliness (7)
- Data Warehouse Testing (6)
- Kanban (6)
- STARwest (6)
- Testing Conferences (6)
- Agile (4)
- Bug Report Attributes (4)
- Don't Test It (4)
- Stareast (4)
- documentation (4)
- Failure Story (3)
- Lightning Talks (3)
- Testing Related Ideas (3)
- You're A Tester (3)
- Performance Testing (2)
- Podcast (2)
- ATDD (1)
- BDD (1)
- HATDD (1)
- Meetups (1)
Who am I?
- Eric Jacobson
- Atlanta, Georgia, United States
- My typical day: get up, maybe hit the gym, drop my kids off at daycare, listen to a podcast or public radio, do not drink coffee (I kicked it), test software or help others test it, break for lunch and a Euro-board game, try to improve the way we test, walk the dog and kids, enjoy a meal with Melissa, an IPA, and a movie/TV show, look forward to a weekend of hanging out with my daughter Josie, son Haakon, and perhaps a woodworking or woodturning project.
My other passion is woodturning.
Test blogs I try to read
Phew, I thought you were having a go at my blog when I saw the title..
isn't one of the expected results "system should not throw an unhandled exception/crash" ?
Damn straight. Run tests find bugs.
there's likely to be less communication overhead though if the expected results are there.
expected results: ?
actual results: x
dev: oh, actual results are x (blah blah blah). i could do j, k or possibly l. i better clarify.
(and dev bugs you right in the middle of your testing)
OR
dev: i'll just apply n. it'll probably be good enough.
(and once aut is back in your hands)
tester: curses! this approach isn't enough (or he should've done m instead)!
i suppose another factor is how smart the dev you're working with is ;)
Philk,
Yes, "system should not throw unhandled error" seems like better expected results than "?". It would raise fewer eye brows.
But if we just use the inverse of the actual results, how much value are we adding? Hmmmm.
Anonymous,
Good points! However, the devs shouldn't be asking testers if devs should fix with j, k, or l.
Some testers will answer that question and some devs will continue coming to testers for solutions. That could be bad.
Most solutions are business questions. And even if they're not, testers are not designers and should not be making such important decisions. Right?
Hear, hear. I felt like I'd grown as a tester the day I decided to just log the bugs instead of spending two hours justifying myself for creating the bug. Nice blog :)